<@U2SG23BPU> thanks I guess the question is more t...

# compiler

[raulraja]

@dsavvinov thanks I guess the question is more technical and we are happy to help bring that support if that is feasible.
At this moment I’m just puzzled by the following:

Any random jar you build with Kotlin writing sources ends up in a .class.
Idea is able to parse that presumably through the metadata annotations or parsing the class structure somehow.
Why doesn’t this work for synthetically generated code which also ends up in a class file? What is different from that bytecode and the one a user types in by hand and is properly recognised by the IDE.

Now for the human side…

For what is worth I’m grateful for everything JB does but the state of meta programming and compiler plugins in Kotlin which has been frequently brought up at conferences and many times publicly does not seem to have a Roadmap or goals for that matter that are publicly known to the community. Instead companies like Google are forking the Kotlin compiler because that is easier since they have control of Android Studio and they have announced a Jetpack Compose, a framework as the future of Android Development which is based on a compiler fork.

At this moment Jetpack Compose relies on a bunch of compiler extensions that are not public or have been modified to fit their needs, (many) I have read the sources and checked with their devs.

Does that mean that all extensions they rely upon will be accepted upstream in the Kotlin compiler?. I’m not sure forking the compiler is good for anyone but the current state of meta in Kotlin is forcing companies to do just this.

It’s not good for everyone when there are multiple versions of the compiler unless they are for KEEPs or enhancing the upstream compiler we all should rely upon.

[dsavvinov]

It’s not good for everyone when there are multiple versions of the compiler unless they are for KEEPs or enhancing the upstream compiler we all should rely upon

Yes, for sure. I’m not directly related to work around Compose, but, as far as I know, there’s some work on upstreaming those extension points, and it also includes discussions about how to make those EPs universally useful, not just something only JetPack can use. But it’s a long road, yes.

[dsavvinov]

Why doesn’t this work for synthetically generated code which also ends up in a class file? What is different from that bytecode and the one a user types in by hand and is properly recognised by the IDE.

First of all, a little disclaimer I should’ve written earlier. We’re treading a really deep waters of the compiler here, or even, pretty much unknown, because as I said above, API hasn’t been designed properly yet. So, we’re almost pioneers here, and take all my answers with a grain of salt — it’s my understanding how the things work, not the ground truth, as I may easily miss somthing 🙂

If you’re depending on such jar, then, I think, it should work, and synthetically generated code shouldn’t be different from anything. In the worst case, you may miss @Metadata for synthetically generated declarations, and they will be seen as Java API.

However, if that’s the sources you’re working with, then it’s completely another story. What is your case?

[raulraja]

We are reworking Arrow meta programming to be solely based on compiler plugins and become automatic.
In this first class I’m generating the kinds emulation for types that have one type argument.

For example. For the data type Option<A> there should be a marker class generated called ForOption.

The issue I have is that in the same file I declare Option<A> I can reference compile and run just fine with references to ForOption but IDEA shows them in red. If I go to a new module that depends on Option and ForOption the compiler frontend bails saying ForOption is not a valid reference.
In general we have to expand with these compiler plugins some marker types but also DSLs like the Optics DSL where we enhance companions with new members.

[raulraja]

So this works and compiles fine:

package arrow.sample

import arrow.sample.ForOption

interface Kind<out F, out A>

sealed class Option<out A>
object None : Option<Nothing>()
data class Some<out A>(val a: A) : Option<A>()

[raulraja]

that import is for a generated class in the same package but IDEA shows that in red

[raulraja]

If I depend on the module with the generated class the following won’t compile:

[raulraja]

import arrow.sample.ForOption

[dsavvinov]

Yeah, so, in those sources your compiler plugin works, and it adds synthetic declaration ForOption, right?

[raulraja]

yes, and I can build and run just fine in the command line

[raulraja]

but if added as dependency it will no longer compile

[raulraja]

the same import that works in my module. I checked the classes are public too.

[raulraja]

but all that is in addition to IDEA never seeing anything that is generated despite having contributed almost the same implementation to the SyntheticResolver, PackageFragmentProvider, IRGeneration etc.

[raulraja]

I was this whole time under the impression that what I was doing in the synthetic resolver adding supertypes and in the package fragment provider was actually to tell IDEA to work since I still have to do the same in the IR phase

[raulraja]

I guess I don’t understand what the synthetic resolver is for then since you can just codegen with IR or the class builder and ASM

[dsavvinov]

You can, but then even in CLI compiler frontend won’t be able to resolve references to synthetically added names and will reject the sources even before backend is launched

[dsavvinov]

At this point, we’re getting into a lot of very low-level technical details and issues, and it’s quite hard to discuss them without specific source code.

If you’re not able to share your work publicly, them I have an another suggestion. We have a kotlinx.serialization plugin (https://github.com/JetBrains/kotlin/tree/master/plugins/kotlin-serialization). Even though its sources are in the kotlin repo, it is intentionally moved in a separate Gradle subproject, so it relatively easy to detach it from kotlin repo. Also, it is known to work 🙂 So, I propose you to start playing with it, gradually changing its sources to what you want to do, probably without whole Arrow-specific logic, just integration with Kotlin. Then, it should be easy a) for you to track where you made a change which broke everything b) for everyone else to see your code, because it will be in a public fork and won’t contain any business logic you don’t want to share.

[raulraja]

Thanks @dsavvinov we can make the work public. I’ll work on cleaning it up and share here

[raulraja]

We really appreciate all your help and insight

[dsavvinov]

I don’t feel like I was really helpful so far, but well, thank you too 🙂

[raulraja]

I wrapped the subscription and usage in a DSL to make it easier to test phases. The DSL supports subscription to all available phases

[dmitriy.novozhilov]

AFAIR if you want create IDE plugin from compiler plugin, you should just add some META-INF to your jar, which describes your implemented extension points
You can look at plugin that I wrote some time ago as prototype as example of that META-INF
https://github.com/JetBrains/kotlin/tree/demiurg906/contracts-plugin-serialization/plugins/contracts/contracts-plugin/resources/META-INF

[dmitriy.novozhilov]

and also you need register your extension in ComponentRegistrar in your plugin (see lines 37 and 38)
https://github.com/JetBrains/kotlin/blob/8af1f156b29b91509bd26423ff68c18225642ef5/plugins/contracts/contracts-plugin/src/org/jetbrains/kotlin/contracts/contextual/ContractsComponentRegistrar.kt#L37

[dmitriy.novozhilov]

After that manipulations, you can use same jar as compiler plugin and as IDE plugin (which, of course, you need to install in IDEA manually)

[raulraja]

Thanks @dmitriy.novozhilov , we'll look into those examples

[raulraja]

@dmitriy.novozhilov I have some questions about the ContractExtension. I see in my local fork that is not yet on master. Is that scheduled to be available on a specific kotlin version for compiler plugins?.

We are interested in contracts but we would like to expand the support also to have smart type casts and not just casts to the same type within the Any? -> Any hierarchy.

In our use case we generate this boilerplate for the higher kind emulation in Arrow for example for the Id type:

class ForId private constructor() { companion object }
typealias IdOf<A> = arrow.Kind<ForId, A>

inline fun <A> IdOf<A>.fix(): Id<A> =
  this as Id<A>

We would like to annotate with a contract the fix function to state whenever there is a value of IdOf<A> that is also a smart cast to Id<A>. This is correct because Id<A> : IdOf<A> and nobody else extends that.

Is this something we could accomplish with the Contract extension?

Essentially if this was done with the contracts API we would do something like:

inline fun <A> IdOf<A>.fix(): Id<A> {
  contract { returns() implies this@fix is Id<A> }
  return this as Id<A>
}

but his@fix is Id<A> is not current available as valid syntax in contracts.

[dmitriy.novozhilov]

contracts-plugin-serialization is a branch with prototype of new kind of contracts that can provide some CFA diagnostics. It was an experiment and there is no plans to add that code to master in 1.4 or earlier. If you interested in, you can take a look at demo repository with those contracts: https://github.com/demiurg906/kotlin-contracts-samples

[dmitriy.novozhilov]

With regard to your question about smartcast, I can't reproduced it. Code below works fine for me (kotlin version 1.3.40). Can you give more detailed example?

interface A<T>

interface B<T> : A<T> {
    fun foo()
}

fun <T> A<T>.requireIsB(): B<T> {
    contract {
        returns() implies (this@requireIsB is B<T>)
    }
    return this as B<T>
}

fun <T> test(a: A<T>) {
    a.requireIsB()
    a.foo() // smartcast
}

[raulraja]

oh yeah, I did not realise about the parens because implies is infix and I thought it was an actual syntax keyword, my bad thanks !