https://kotlinlang.org logo
Join Slack
Powered by
How can I resolve the type of a KtProperty inside ...
# arrow-meta
r
How can I resolve the type of a KtProperty inside a meta plugin? I'm trying to do something like this but I don't know if I'm even calling the right API
Copy code
val Meta.demoPlugin: CliPlugin get() = "Demo" {
    meta(
        property({ true }) { ktProperty ->
            val type = ktProperty.typeReference?.name // name is null
            Transform.empty
        }
    )
}
m
At that stage, you can only use the 
text
property. The actual type is unknown, yet, and so its actual name. The only thing that is known is the piece of source code used to identify that type.
r
So how can you get a fully qualified name? Do you need to search through the imports manually?
m
I find that often naked AST are not the most useful material to manipulate when doing metaprogramming. I still don't know how to access the later stages. See: https://kotlinlang.slack.com/archives/CJ699L62W/p1588792167128900
Yes, you are bound to duplicate the work of the compiler's type checker.
Otherwise, you can do as I'm doing right now: copying the imports from the original source file, hoping that nothing breaks. But it's just a temporary hack, limited in its usefulness, while I learn how to use the annotated AST.
"Do you need to search through the imports manually?" You can do that but if there is a "import with wildcard", you would have to inspect all the dependencies and see what they provide under that package, in fact duplicating a big part of the work of the type checker. Also what if you have a delegate property? Etc.
r
Oof yeah maybe I need to re-evaluate my approach
m
Please let me know if you understand how to access the successive stage of compilation or if you find any examples.
r
the next stage is analysisCompleted where you have the tree and descriptors
Before codegen
There is a meta hook for all phases
In that phase you can analize the entire module after it has been typechecked
r
Can you point to any samples? I've only explored things based on the hello world sample and I don't understand how to configure other phases
r
https://meta.arrow-kt.io/apidocs/compiler-plugin/arrow.meta.dsl.analysis/-analysis-syntax/analysis.html#arrow.meta.dsl.analysis.AnalysisSyntax$analysis(kotlin.Function7((arrow.meta.phases.CompilerContext,%20org.jetbrains.kotlin.com.intellij.openapi.project.Project,%20org.jetbrains.kotlin.descriptors.ModuleDescriptor,%20org.jetbrains.kotlin.context.ProjectContext,%20kotlin.collections.Collection((org.jetbrains.kotlin.psi.KtFile)),%20org.jetbrains.kotlin.resolve.BindingTrace,%20org.jetbrains.kotlin.container.ComponentProvider,%20org.jetbrains.kotlin.analyzer.AnalysisResult)),%20kotlin.Function5((arrow.meta.phases.CompilerContext,%20org.jetbrains.kotlin.com.intellij.openapi.project.Project,%20org.jetbrains.kotlin.descriptors.ModuleDescriptor,%20org.jetbrains.kotlin.resolve.BindingTrace,%20kotlin.collections.Collection((org.jetbrains.kotlin.psi.KtFile)),%20org.jetbrains.kotlin.analyzer.AnalysisResult)))/analysisCompleted
You can add those functions to your meta block
And it will call you each time it goes through it
The binding context and trace are available and there is the relationship between all psi and their descriptors
You can regenerate sources there and reanalize if you want but it's slower than a multiphased approach since you will be effectively rewinding and reparsing
In the IDE you can analize psi directly
These DSL phases only work in the compiler , if you want anylisis on highlighting you can analize nodes individually but that only works on idea
Which has done a previous typechecking phase for highlighting
And it retriggers it each time you type
m
Thanks, Raul. It's very useful for me, too.
r
Yeah thanks Raul. I'll dig in more tomorrow.
r
No problem, if you have specific questions on where to get things ask away, I'm familiar with those internals and you can accomplish the same thing in many ways
r
Still stuck here. I'm grabbing the 
KtProperty
reference I want in a 
property()
call, then doing an 
analysis()
call to try and check that property's type. Not sure what the correct API to call is, but all of the below are still giving me null inside the 
analysisCompleted
lambda (and I don't know if any of them are what I should be doing anyway):
Copy code
bindingTrace.getType(ktProperty)
bindingTrace.bindingContext.getType(ktProperty)
ktProperty.kotlinType(bindingTrace.bindingContext)
Should also note I'm just returning 
null
in the 
doAnalysis
lambda which might also be causing issues. What's the correct default to use there to skip straight to 
analysisCompleted
?
r
returning null is correct there because it’s what the compiler expects which means you have not interrupted the compilation
you could also return error in analysis
the ktproperty to descriptor relationship can be seen inside the binding context map and extracted by the keys for descriptors
map psi elements to what the typechecker and analysis now about the program before codegen
so you can just extract the pairs getting the value of the slices and iterating
you can extract the binding context from the binding trace which is received in the analysis function
calling analyze on a psi element directly only works in the IDE
also in the CLI you can’t call replace or methods that mutate the AST in place as its done in the IDE
in the IDE the tree is mutable and in the cli is immutable
r
Any suggestions for which slice to use? Most of my attempts to call 
bindingTrace.bindingContext.getKeys(<http://BindingContext.XXX|BindingContext.XXX>)
are throwing with "Keys are not collected for slice XXX"
r
r
If I call 
bindingTrace.bindingContext.getKeys(BindingContext.TYPE)
I get 
java.lang.AssertionError: Keys are not collected for slice TYPE
r
can you try iterating directly over the slice?
not using getKeys
I’ve tried this for Calls and other things and has always worked
that one has all the descriptors associated to Psi and the descriptors is where all the typechecked tree is for a given node
r
I'm not clear what you mean. What am I iterating over?
I don't see where I get an iterable from 
WritableSlice
r
let me fire up meta
r
If it helps, this is a stripped down version of what I'm trying to do:
Copy code
val Meta.demoPlugin: CliPlugin get() = "Demo" {
    val properties = mutableSetOf<KtProperty>()
    meta(
        property({ true }) { ktProperty ->
            properties += ktProperty
            Transform.empty
        },
        analysis(
            doAnalysis =
            { project: Project,
              moduleDescriptor: ModuleDescriptor,
              projectContext: ProjectContext,
              collection: Collection<KtFile>,
              bindingTrace: BindingTrace,
              componentProvider: ComponentProvider ->
                null
            },
            analysisCompleted =
            { project: Project,
              moduleDescriptor: ModuleDescriptor,
              bindingTrace: BindingTrace,
              collection: Collection<KtFile> ->
                properties.forEach { ktProperty -> 
                    // Looking for the type of ktProperty here
                }
                null
            }
        ),
    )
}
r
type information is only available on `analisysCompleted`though you can hijack that immutable map with reflection in 
doAnalysis
and have raw power to whatever you want to do there
r
Ah ok that helps a lot. 
getSliceContents()
is what I was missing
Once I have the type information I'm looking for, I will want to generate some new code. Does that mean I'll need to do it in 
doAnalysis
? Or can I call one of the quotes APIs next and have that run after?
r
Quotes happen in doAnalysys. Never tried rewinding there but you may try returning the análisis result that retries with additional roots
If you can get this to work we can generalize it as a pattern for quotes that need type info and rewind
r
How do I rewind?
r
r
ok I'll see what I can figure out
I do think this would be generally useful functionality to have, for a lot of traditional annotation-processing use-cases
👍 1
r
Makes sense
r
Ok yeah that retry seems to work. Surprisingly I didn't even need to add any logic to guard against hitting it again on the retry
r
right so if you retry with the binding context the second time you go through quotes you may have type info directly associated in the quotes
I believe you are retrying with a context that has already type-checked those elements and you are providing new ones as source so it may skip typechecking of what has already been done unless you mutate in place
but it shouldn’t if all new sources go to new files
since the KtFile has already been marked as analyzed in the binding context
r
ok. Yeah currently I'm just changing existing source but my next step is generating new stuff so I'll keep an eye out for how that breaks
r
if that works we have a great new feature with meta which is just exposing the correspondent type descriptor and auto rewind for all quotes that need type info and not just structure
this has been a major blocker and something they are gonna improve in FIR I believe in the compiler so at least you have a multiphased approach to transforming the tree while in the frontend
r
I'll keep messing around and let you know how it goes. Thanks a ton for all the help!
r
likewise, this is helping meta too 🙂
@bloder this may interest you. There may be a way to make Transform.newSource or quotes in general get type info alongside the tree info after the first analysis
b
just to confirm if I got it correctly: You mean make our quote system gets the updated ast after any kind of modification in its plugin?
r
Since the quote plugin runs on doAnalisys
you can return RetryWithAdditionalRoots
as result of analysis passing the already typechecked trace and binding context to that method
r
The behavior is actually getting less clear to me as I mess around with it. It seems to work sometimes and not others. Will try to get a minimal sample together that you guys can check out
r
thanks @russhwolf
b
Hmmm an example with that would be nice to help me to understand what we are looking for! I'm wondering how it should look like, in my mind I'm seeing something like:
Copy code
...
meta(
analysis(...) + namedFunction({ name == "helloWorld" }) { c -> }
)
...
And maybe this is what I was looking for some time ago when I've created a lib with an old version of meta, compose extension phases would fix a workaround that I did in that project, but I'm not sure if is that what we are looking for here in this case
r
Ok. I think it maybe only looked like it was working because some state is getting cached somewhere in-between runs. I'm filtering for some KtProperties and storing them in a list inside the plugin block. Then in analysis I try to get the fully-qualified type of those properties and cache those in another list. I next want to modify those types via quotes. When trying this repeatedly, these caches are not being cleared between runs which makes it look like it's inconsistently possible to pass information back to quotes constructs from an analysis block
But with enough logging in place it's pretty clear that I'm never seeing my quotes blocks actually run after analysis
b
I don't know exactly where it's located this inconsistency but I don't think we have a good support for multi extension phases today and probably this is the reason that is happening all those caching between them with no supervising, please @raulraja correct me if I'm wrong, but transform the phases into kind of monads would help with it since in the end all extension phases composition using RetryWithAdditionalRoots would result in a single phase with no or supervised caching between its composed phases
Similar to what we did with our transforms
r
that could work but also we can just expose the corresponding descriptor as nullable in the tree
so nothing changes. For example 
Property.descriptor?
r
Back to messing around here, and I'll work on putting a useful example together, but it's looking like if you take the 
processFiles()
and `updateFiles()`calls that the quotes system uses internally in 
doAnalysis
, and move them to 
analysisCompleted
with a 
RetryWithAdditionalRoots
return, then things work.
r
that is promising thanks @russhwolf, if you can put a small example of a plugin that does that we can derive the pattern and include in meta automatically so you get the types right on the quotes
If you are interested I can guide you on how that can be contributed and I’m sure that @bloder may also be able to help
👍 1
a
I'm also very interested in this, when can we expect this? Is there anything I can help with?
b
@Ahmed Mourad Im just waiting the sample to confirm that I understand what is proposed here and how to do it and I'll start work on that or if you or @russhwolf would like to contribute with that I can guide you.
arrow 1
a
Sounds great, I'm in!
Are there any updates on this? or any workaround yet?
r
It has not been built yet as part of meta. I asked @Imran/Malic to introduce it as part of his quote changes. The ability for meta quotes to contain type information using RetryWithAdditionalRoots
arrow 1
💯 1
a
Cool! Is there an issue on Github to follow to know when it's ready?
i
I will send in a PR, when I have something working. 🙂
arrow 1
Me and @Joachim Ansorg were working on fixing a few bugs in the quote system on the Ide side, first.
arrow 1
a
Alright then,, I'll keep an eye out for PRs, keep on the awesome work! 🙂
d
Hey @russhwolf, did you manage to put a working example? I'm struggling with the same issues as you
r
@bloder is looking to see if this can be added to Meta automatically to have typed quotes
b
I`ll take a look and study a solution for that today, once is ready I`ll open a PR / issue with my proposal and I share with you
🙌 3
🔥 1
d
Is there an issue for this on github so we can follow? On another topic - I'm still playing around and getSliceContents always returns an empty map which isn't consistent with the example given above
r
@dephinera where are you calling getSliceContents?
d
in analysisCompleted
r
that should work, can you post the snippet where you call it?
b
Hello guys I`m taking a look to try to map this error correctly and I`m simulating this thread property sample, I`ve created a similar sample and I just can`t get the 
name
property but Quotes has its type reference and I can get kt property reference by 
text
property:
Copy code
meta(
        property(this, { true }) { ktProperty ->
          messageCollector?.report(CompilerMessageSeverity.ERROR, "KtProperty - ${ktProperty.name} - ${ktProperty.typeReference?.name}")
          messageCollector?.report(CompilerMessageSeverity.ERROR, "KtProperty ${ktProperty.name} - ${ktProperty.typeReference?.text}")
          messageCollector?.report(CompilerMessageSeverity.ERROR, "Quote - ${this.name} - ${this.typeReference}")
          Transform.empty
        }
      )

val x: Repository = Repository()
class Repository

// e: KtProperty - x - null
// e: KtProperty x - Repository
// e: Quote - x - Repository
could you provide me what I am missing about it, please? Because I`m seeing here that Quotes and its KtElements provides a typed info in analysis phase
but not with name property but with text or toString in case of quotes
r
it’s missing having the associated descriptor to each quote
which can be extracted from the binding context before returning RetryWithAdditionalRoots and passing back the files to reanalize
👍 1
b
@dephinera I`m with the same problem with slices api that ure having, declaration to descriptors is always returning an empty map for me but fqname to class descriptor give me my type descriptor
Copy code
analysis(
          doAnalysis = { project, module, projectContext, files, bindingTrace, componentProvider ->
            messageCollector?.report(CompilerMessageSeverity.WARNING, "PASSED HERE!")
            null
          },
          analysisCompleted = { project, module, bindingTrace, files ->
            val slice: ImmutableMap<PsiElement, DeclarationDescriptor> =
              bindingTrace.bindingContext.getSliceContents(BindingContext.DECLARATION_TO_DESCRIPTOR)
            messageCollector?.report(CompilerMessageSeverity.WARNING, "PASSED HERE22! -- ${slice}")
            slice.forEach { (psi, descriptor) ->
              if (psi is KtProperty && descriptor is PropertyDescriptor) {
                messageCollector?.report(
                  CompilerMessageSeverity.ERROR,
                  "${psi.text} -> Type: ${descriptor.returnType}"
                )
              }
            }
            AnalysisResult.RetryWithAdditionalRoots(bindingTrace.bindingContext, module, additionalJavaRoots = listOf(), additionalKotlinRoots = listOf())
          }
        )
this is my code that I`m analysing
Copy code
val x: Repository = Repository()

class Repository

fun main() {
  println(x)
}
d
@raulraja My sample is pretty much the same as @bloder's.
Copy code
val Meta.example: CliPlugin
    get() =
        "Example" {
            meta(
                    analysis(
                            doAnalysis = { project, module, projectContext, files, bindingTrace, componentProvider ->
                                null
                            },
                            analysisCompleted = { project, module, bindingTrace, files ->
                                val slice: ImmutableMap<FqNameUnsafe, ClassDescriptor> = bindingTrace.bindingContext.getSliceContents(BindingContext.FQNAME_TO_CLASS_DESCRIPTOR)
                                messageCollector?.report(
                                        CompilerMessageSeverity.WARNING,
                                        "slice empty: ${slice.isEmpty()}"
                                )
                                null
                            }
                    )
            )
        }
I can see that he experienced the same problem. @bloder can I extract properties information from FQNAME_TO_CLASS_DESCRIPTOR? I'm playing around with the information from it but I can't find an api that will do the job
Okay, I just managed to extract property's information using 
FQNAME_TO_CLASS_DESCRIPTOR
Copy code
analysisCompleted = { project, module, bindingTrace, files ->
    val slice: ImmutableMap<FqNameUnsafe, ClassDescriptor> = bindingTrace.bindingContext.getSliceContents(BindingContext.FQNAME_TO_CLASS_DESCRIPTOR)
    slice.forEach { (fqName, descriptor) ->
        descriptor.getMemberScope(TypeSubstitution.EMPTY).getDescriptorsFiltered { true }.forEach {
            if (it is PropertyDescriptor) {
                messageCollector?.report(
                        CompilerMessageSeverity.WARNING,
                        "property name: ${it.name}"
                )
                messageCollector?.report(
                        CompilerMessageSeverity.WARNING,
                        "property return type: ${it.returnType}"
                )
                it.annotations.forEach {
                    messageCollector?.report(
                            CompilerMessageSeverity.WARNING,
                            "property annotation: ${it.fqName}"
                    )
                }
            }
        }
    }
    null
}
There is something I don't get though. If returning 
RetryWithAdditionalRoots
needs new files in order to work and skips the already analyzed files, how are we going to perform modifications on the already analyzed files. @bloder what's your plan to apply modifications after you resolved the types in 
analysisCompleted
?
Update: I think I experienced the same thing that @russhwolf did. It appears there some sort of caching as sometimes my plugin works and sometimes it doesn't. I have two phases - property (quote) and analysis. I'm storying a simple <String, KotlinType> map in the project file and I'm populating it in 
analysisCompleted
. There I generate a dummy file, which I pass to 
RetryWithAdditionalRoots
. In the property phase I apply a transformation only if the property name is present in the map. The transformation itself is replacing the property with itself and another one. The behavior is not consistent from run to run. It appears that it works every second run but I wouldn't rely on that. Another thing that I'm observing is that all my messages from all phases are printed only once.
b
@dephinera Well, an update about that, I actually already did the rewind of quotes in analysisCompleted I'm having some problems with our test library but I'll fix later and I'm studying this solution that is basically let quotes apply file transformations in analysisCompleted and returns RetryWithAdditionalRoots passing an empty new file list but the root files are already transformed by quotes at that time then it just perform the analysis phase again with already transformed files, creating quote test internal files, I'm also studying a possibility of let this completed phase binds only the quote descriptor and that's what I'm working now.
And about descriptors I could find my descriptors by calling that array of writable slices that binding context provide us, once I'm on my computer I'll send u
d
Good to hear. I'm looking forward to seeing your work
👍 1
b
@dephinera This is how I`m getting all descriptors from the analysed phase
Copy code
BindingContext.DECLARATIONS_TO_DESCRIPTORS.flatMap {
          it.makeRawValueVersion().let<ReadOnlySlice<Any, Any>, ImmutableMap<Any, Any>>(bindingTrace.bindingContext::getSliceContents).toList()
        }
👍 1
r
Hey all sorry I never got my code sample uploaded anywhere. Sounds like you’re making some progress though. The PoC I was doing was specific to delegated properties so I ended up making it work with 
bindingTrace.bindingContext.getSliceContents(BindingContext.DELEGATED_PROPERTY_RESOLVED_CALL
. I was never able to read anything out of 
DECLARATIONS_TO_DESCRIPTORS
but I had some stuff sort of working using the 
TYPES
slice. Problem there was I didn’t always get a fully-qualified name.
b
Hi! What I`m doing here is use 
DECLARATIONS_TO_DESCRIPTORS
that gives me all descriptors from analysed code and pattern matching them with quotes, I`m creating right now a new type of quote that support descriptors, once I have an example I can show you, there are tests missing yet, but I think I`m making progress here 🙂
d
hey, @bloder, nice work and thanks for sharing. Do know however what caues that inconsistent behavior when I try to save info in analysisCompleted and sometimes it's present after rewind and sometimes not? How are you actually implementing the new quote api, are you still rewinding or are you just calling that api directly from analyssicompleted without the need of rewinding?
b
I'm rewinding but I didn't face inconsistency how are u saving your info?
d
Well for the time being I created a map property in the Kotlin file and I'm populating it in analysisCompleted. Sometimes it's populated, sometimes it's not
let me gather an example
Okay, that's embarrassing.. ;d I just figured that inconsistency out. I'm relying on that 
analysisCompleted
will be called twice and I will remove the dummy file I created during the second run, while it's actually called once. There's a check that's basically does 
if (file.exists) file.delete
so therefore every second run it's working, because the previous one has deleted the file... I still don't get why 
analysisCompeted
seems to be called only once. Should it be called twice due to rewinding?
r
I think the rewinding makes this look weird. I was printing logs to verify that I was detecting the things I expected. On successful runs those logs would be empty even though the outputs made it clear it had found stuff. But if I threw an exception at the end I'd get the logs I expected. It seemed like when it retried certain things were getting cleared out in a way that made it look like the first pass hadn't happened, even though changes made during that pass still were present.
d
so it's like the message collector just keeps the last messages that happened per phase
r
That's what it seemed like. I don't really understand why you'd want that behavior, though.
👍 1
10 Views
Open in Slack
PreviousNext
Powered by