https://kotlinlang.org logo
Join Slack
Powered by
Sorry if this is an obvious question, but where is...
# arrow-meta
c
Sorry if this is an obvious question, but where is 
arrow-reflect-compiler-plugin
published? I'm attempting to use it from a build system that isn't gradle, but I can't find it in maven. Tangentially, is using 
arrow-reflect
recommended right now, or should I use 
arrow-meta
directly?
r
Hi @conner, arrow-reflection is not published, we stopped development until K2 was finally stable as we were chasing the api changes and found several blockers during its development. May be ok now to pick up again. As for 
arrow-meta
is usable if you are gonna do anything with just kotlin backend IR, it has not been updated to latest versions of kotlin. We want to pick it up and finalize arrow-reflection as to what arrow-meta will finally become. A simple library for macros in kotlin. There are still a lot of rough edges and corners in there that don't work like they should due to limitations with the kotlin compiler api.
c
Thank you for the context!
a
for Kotlin 1.9.20-Beta 2 and Arrow-Meta there are only a few changes for the compiler (
ClassBuilder
uses a deprecated class, needs to become extension on 
ClassGeneratorExtension
I think, but the error gives another options as well) references now use 
IdSignature
and to use descriptors you need to go down a level, so:
Copy code
pluginContext.symbols.externalSymbolTable.referenceField(this)
becomes
Copy code
pluginContext.symbols.externalSymbolTable.descriptorExtension.referenceField(this)
Easy so far. BUT, then 
kotlin-compiler-testing
is so far out of date that all tests cannot be run (calls 
useIr
on compiler args but that is no longer a valid argument). PR's for 
kotlin-compiler-testing
for updating to 1.8.2x and then 1.9.x are all stalled it appears.
For Arrow-Reflection there are some small additions of a new context parameter to 
FirMetaCodegenExtension.kt
. A now missing 
FirQualifiedAccess
class used from Meta.kt that probably should replaced with something but it isn't used yet anyways. And then all the work to upgrade is in 
TemplateCompiler
with 
Fir2IrResult
becoming 
Fir2IrActualizedResult
and a bunch of related changes in 
convertToIR()
. Some new configuration is required and the call for 
convertToIrAndActualize
I think becomes 
convertToIrAndActualizeForJvm
unless you really want to deal with 10 new parameters that are fun to configure. But I'm not sure all of the intent there. I did this just to get compiling to see where things stood. Add
Copy code
configureFirParser(FirParser.Psi)
to the base test runner and you can get moving. But, I tried compiling the sandbox and get errors about FIR cache being cleared blowing everything up. tests all fail because of different generation, but it must be close. but sandbox blows up:
Copy code
Unexpected caches clearing
        at org.jetbrains.kotlin.fir.resolve.providers.impl.FirCachingCompositeSymbolProvider.createCopyWithCleanCaches(FirCachingCompositeSymbolProvider.kt:77) 
...
TemplateCompiler
anyways, If I was confident in my changes i could push a few PR's.
Of interesting note that Intellij common built for k2 with compiler plugin loading enabled, does show the sandbox call to 
product()
as a non-error and the correct type from the generated code.
Screenshot 2023-09-25 at 7.53.34 PM.png
let me know if you want me to send the dirty PR for reference.
r
@apatrida thank you for looking into all this. We were planning on picking up 
arrow-reflection
once things were stable. We'd very much appreciate those PRs/suggestion to get us closer to update the project. thank you color
a
@raulraja here is Meta to 1.9.20-Beta2 to show where it stands (will check 1.9.30-dev-3330 which is the current dev version in a bit) https://github.com/arrow-kt/arrow-meta/pull/1075
thank you color 1
1.9.30-dev-3330
looks to be the same, no changes for compiling and tests passing.
r
All changes there make sense, thanks for looking into it!
a
A problem in arrow-reflection and 1.9.20-Beta2 is a hack added to 
FirCachingCompositeSymbolProvider
with comments:
Copy code
// This property is necessary just to make sure we don't use the hack at `createCopyWithCleanCaches` more than once or in cases
    // we are not assumed to use it.
    private val expectedCachesToBeCleanedOnce: Boolean = false,
and
Copy code
// Unfortunately, this is a part of a hack for overcoming the problem of plugin's generated entities
    // (for more details see its usage at org.jetbrains.kotlin.fir.resolve.transformers.plugin.FirCompilerRequiredAnnotationsResolveProcessor.afterPhase)
    fun createCopyWithCleanCaches(): FirCachingCompositeSymbolProvider {
        require(expectedCachesToBeCleanedOnce) { "Unexpected caches clearing" }
        return FirCachingCompositeSymbolProvider(session, providers, expectedCachesToBeCleanedOnce = false)
    }
Which explodes when 
TemplateCompiler
calls 
afterPhase
I'm looking to see how this is supposed to behave and what is done here that causes the exception. The move from 1.9.20-Beta to 1.9.30-dev-3330 is a bit more involved, so I'm doing them separately.
oh, the above comment says to read this:
Copy code
@OptIn(FirSymbolProviderInternals::class)
    override fun afterPhase() {
        super.afterPhase()

        val generatedDeclarationsSymbolProvider = session.generatedDeclarationsSymbolProvider
        generatedDeclarationsSymbolProvider?.enable()

        // This part is a bit hacky way to clear the caches in FirCachingCompositeSymbolProvider when there are plugins that may generate new entities.
        // It's necessary because otherwise, when symbol provider is being queried on the stage of compiler-required annotations resolution
        // we record incorrect (incomplete) results to its cache, so after the phase is completed we just start from the scratch
        val symbolProvider = session.symbolProvider
        if (generatedDeclarationsSymbolProvider != null && symbolProvider is FirCachingCompositeSymbolProvider) {
            @OptIn(SessionConfiguration::class)
            session.register(FirSymbolProvider::class, symbolProvider.createCopyWithCleanCaches())
        }
    }
@raulraja the other PR for reflection: https://github.com/arrow-kt/arrow-reflection/pull/36 this fails that assert mentioned above, I'm still looking into it. Also to go from there to 1.9.30-dev-3330 is a bunch of changes that need researched through the compiler git history as well. I'm looking at those also.
thank you color 1
r
Those all make sense as well, thank you! . The Template Compiler included a bunch of internals ported from the compiler, not surprised is broken in so many places. Ideally parsing an expression in the context of a current FirSession and producing IR for it would be something the compiler offered directly but we had to hack it to get some form of annotation macros working. When we are at a good place we'd like to bring what is in arrow-reflection to meta and then Arrow Reflection + IR support is what meta ends up being.
a
I started to compare Template Compiler to where it came from, a few changes in there were due to that diff (minor things), but that's when I noticed it came from there. I'll check the sources for those I can identify and see if that helps. The other changes I traced somewhat and look ok conceptually and I just have to do them to see what others breaks cascade from them... Bit by bit.
Most of the 1.9.30-dev-3330 level changes are around 
typeRef
becoming 
resolvedType
(or 
coneTypeOrNull
returning 
ConeKotlinType
Raul, was 
CompilerPipeline.kt
there when you started, with methods such as 
transformFirToIr
?
(added March 7, 2023)
probably was as 
convertToIrAndActualize
is in that file
r
or an earlier version of that which is the one in 1.8.20
a
great, then I can track changes to see what are material changes since
Rual, for 1.9.30-dev-3330 which is most current compiler https://github.com/arrow-kt/arrow-reflection/pull/37 this includes the previous changes, the delta can be seen in the last commit: https://github.com/arrow-kt/arrow-reflection/pull/37/commits/d8a31f14e0e1f6dc11989985c49988a21fd44d09 This still has the same failure, which is what I'm looking at now, how to resolve. ("unexpected caches cleared")
uh oh 
build-2.0.0-dev-4182
... things are moving!
ok, so you pulled from: 
FirTotalResolveProcessor
, 
ConvertToIr
, 
compilerPpeline
, and in there is the reference to 
FirCompilerRequiredAnnotationsResolveProcessor
which has the code that calls the 
FirCachingCompositeSymbolProvider::createCopyWithCleanCaches()
method that blows up, so I'm checking for differences in the code you pulled THEN to NOW
Have a temp work around, since they are wanting to clear possibly inconsistent caches, I can just clear them ahead of time instead of passing in an already cleared-1-time cache which it dislikes.
Copy code
// in TemplateCompiler.kt

@OptIn(SessionConfiguration::class)
    fun process(files: List<FirFile>) {
      for (processor in processors) {
        // clear the cache myself
        if (processor.session.symbolProvider is FirCachingCompositeSymbolProvider) {
          processor.session.register(FirSymbolProvider::class,
            FirCachingCompositeSymbolProvider(processor.session,
              (processor.session.symbolProvider as FirCachingCompositeSymbolProvider).providers, true)
          )
        }

        processor.beforePhase()
        when (processor) {
            ...
this gets things running again, I'll test it more in Intellij-community with compiler plugin loaded to see if other errors come up. Other tests fail due to codegen changes, but not focused on those if things appear to work.
Just not sure the rules yet (waiting on a reply related to this). Like if we clear these caches, does that mean it won't see things it needs to from the first compile to do the additive codegen compile? I can always wrap the cache with a cache to clear the wrapper but not the real cache as well. Will get more info. Getting ASM9 error at the moment.
Copy code
PermittedSubclasses requires ASM9
	at org.jetbrains.org.objectweb.asm.ClassVisitor
think related to JDK version running needs to be older, tracking that and other issues...
will come back to this one:
Copy code
org.jetbrains.kotlin.utils.exceptions.KotlinIllegalArgumentExceptionWithAttachments: Expected FirResolvedTypeRef with ConeKotlinType but was FirUserTypeRefImpl
	at org.jetbrains.kotlin.fir.types.FirTypeUtilsKt.getConeType(FirTypeUtils.kt:192)
	at org.jetbrains.kotlin.fir.resolve.ResolveUtilsKt.getCorrespondingClassSymbolOrNull(ResolveUtils.kt:507)
	at arrow.reflect.compiler.plugin.fir.checkers.FirMetaAdditionalCheckersExtensionKt.metaAnnotations(FirMetaAdditionalCheckersExtension.kt:90)
	at arrow.reflect.compiler.plugin.fir.codegen.FirMetaCodegenExtension.nestedClasses(FirMetaCodegenExtension.kt:190)
	at arrow.reflect.compiler.plugin.fir.codegen.FirMetaCodegenExtension.getNestedClassifiersNames(FirMetaCodegenExtension.kt:198)
info:
Copy code
- typeRef:
    Class: org.jetbrains.kotlin.fir.types.impl.FirUserTypeRefImpl
Value:
    Product

========
- typeRefElementKind:
    KtRealSourceElementKind
========
- typeRefSource:
    Class: org.jetbrains.kotlin.KtRealPsiSourceElement
Value:
    <File name: Sample.kt, Physical: true>
@<ELEMENT>Product</ELEMENT>
data class Sample(val name: String, val age: Int)
so the annotation itself and how it is typed is a problem.
pushed to PR branch, will come back to continue on these issues later. any ideas welcome, otherwise will just trace through everything to see how these are created and why they error here.
The failures above do not happen in unit tests (generated box nor diagnostic), they only fail due to different ordering and content in the generated fir.ir ... so using the plugin in another instance of the IDE and it's behavior causes things to go awry.
suddenly:
Screenshot 2023-09-26 at 4.45.26 PM.png
but things are broken in the code even though it is roughly working, we lose the ability for that 
main
to work
Non-static main() method was invoked: class must have constructor with no parameters
9 Views
Open in Slack
PreviousNext
Powered by