https://kotlinlang.org
Join Slack
If I generate a list of items per-module in an Android project, is there a way I can consolidate all...
# ksp
c
If I generate a list of items per-module in an Android project, is there a way I can consolidate all those lists into a single, generated set with KSP?
m
Are your items Kotlin source files?
c
No, they’re just 
KClass
instances
a list of KClasses per module
m
So you would run KSP on your own KSP-generated code?
I think that could work. Don't want to say something stupid but IIRC, KSP looks in dependencies
c
Interesting… currently, I’m generating the 
List<KClass<*>>
in a file in each module and combining them manually
I haven’t seen KSP work on my dependencies
m
module1:
Copy code
package module1

@MyAnnotation
val classes = listOf(Foo1::class, ...)
module2:
Copy code
package module2

@MyAnnotation
val classes = listOf(Foo2::class, ...)
aggregatingModule processor:
Copy code
getSymbolsWithAnnotation(@MyAnnotation)
c
Ohhhhhh
m
Not 100% sure it works, try it out
c
I’ll give it a try, thanks
m
If it's easier, you could also generate json side metadata in resources and then aggregate that without even using KSP in your aggregating module
Copy code
codeGenerator.createNewFile(
  Dependencies.ALL_FILES, // or proper dependency tracking
  "",
  "META-INF/myprocessor/classesList.json",
  ""
)
c
I’m fairly new to KSP, so I’m not precisely sure what that does, or how to use it exactly
m
Most of the time you use KSP to generate additional Kotlin source code (your classes list above)
But if you want to re-process that, sometimes Kotlin is not the best interchange format because you need to parse it again (what KSP is doing)
And JSON data is usually easier to parse than Kotlin data
Both could work. With the 
getSymbolsWithAnnotation(@MyAnnotation)
route you'll have to use KSP APIs to read 
Foo1
, 
Foo2
, etc.. from the Kotlin AST.
Thinking more about this, 
Foo1
, 
Foo2
, etc... is inside the initializer and I don't think KSP gives you access to this
KSP is only about the signatures, not the code
It gives you "constant" code so you could maybe duplicate the list:
Copy code
package module1

@MyAnnotation(Foo1::class, Foo2::class, ...) // This is constant
val classes = listOf(Foo1::class, ...) // This is not
Depends what you want to generate ultimately
c
I was under the impression that since KSP runs per-module, I wouldn’t be able to generated code that lives in other modules.
m
I'm curious now, let me try quickly
https://github.com/google/ksp/blob/69060b25dcaa98a9c7a3641aa0e7545a20b94985/api/src/main/kotlin/com/google/devtools/ksp/processing/Resolver.kt#L48 There's a promising 
inDepth
parameter 👀
Copy code
fun getSymbolsWithAnnotation(annotationName: String, inDepth: Boolean = false): Sequence<KSAnnotated>
c
Hmm I can try that real quick
Doesn’t seem to do anything differently in my case
I was having the problem earlier with my processor running for each module, and overwriting the file that the previous module generated, so I had to pass a unique identifier for the file name in each module’s ksp args. So now I’m left with several generated lists across several modules.
This is interesting. I hadn’t considered plugins. https://stackoverflow.com/a/78387425/3692626
didn’t even know there were plugins
d
There are KSP processors that aggregate over modules. I think https://github.com/airbnb/Showkase is an example and it did it using the 
resolver.getDeclarationsFromPackage(packageName: String)
API which gets from dependencies and source.
👀 1
👍 1
c
Oh nice
d
They may have changed the approach, so might be worth checking.
c
Let me try that out. Thanks.
m
There's a promising 
inDepth
parameter
Can confirm 
inDepth
doesn't look in dependencies
getDeclarationsFromPackage
it is 👍 til
c
Thanks! I’ll let you know if I have any success 🙂
👍 1
Doesn’t seem like 
getDeclarationsFromPackage
will work for my scenario. It requires the exact package name, and these classes that I’m getting declarations for have no fixed package :(
I suppose I could aggregate them all in our codebase 🤔
d
getSymbolsWithAnnotation
will work for the processor that runs on the individual modules right? This processor could write Kotlin files for the individual modules and set the generated code's package name to something fixed for your entire project. Then the processor that runs on the top-level module can use 
getDeclarationsFromPackage
to pick up the outputs with the fixed package name.
➕ 1
m
Agree with David here, if you use the same package name everywhere it should work (or just write a json resource if you want different package names)
Also, filed https://github.com/google/ksp/pull/2136 for the record
c
Awesome, thank you!
Would I need to create a separate module for the final round processing that aggregates the values from the other modules, and only use 
ksp(project(:finalround))
in that main module? I don’t want each module to trigger the aggregation. This might be a stupid question, but does it need to be a separate processor?
m
You can reuse the processor and pass it an argument
c
I thought that might be the case. I think I have a pattern in mind, but any examples of this being done are also appreciated 🙂
m
I use args in gratatouille. • The arg is set here • And read here
c
Thanks - I was more looking for multi-round examples
Is there a common pattern or means for terminating multi-round processing? I thought I could just detect if the file exists already via 
resolver.getAllFiles()
with 
ksp.incremental=false
, but by the last invocation of the processor, 
resolver.getAllFiles()
doesn’t contain my generated files, and it tries to start from the beginning. Probably something dumb on my end.
And in that last invocation, none of my ksp option args are present either
If that’s by design, I could use the absence of the options to terminate, I suppose
Is there some mechanism to determine when all contributing modules have generated their code, and the main module should proceed with aggregating them?
z
you would need to check for an elements annotated with their annotations in your current round and defer your elements if any are present
👍 1
Also not sure this needed bumping back into the main channel when you already had people helping in this thread 🙂
c
For the aggregation step, I’m using 
resolver.getDeclarationsFromPackage
, where the package is the same for all modules that generate code. I’m using 
resolver.getNewFiles
to see if any new files were created, and if they were, I defer the symbols I obtained from 
getDeclerationsFromPackage
, but no new rounds are triggered for me to attempt aggregation again.
I think perhaps there is no real way to know when all contributing modules are done generating code, just using KSP, so that aggregation may continue. Maybe the aggregation step isn't a job for KSP :\
m
If it’s different Gradle modules, shouldn’t Gradle ensure ksp tasks are run sequentially? • module1kspKotlin • module2kspKotlin • : aggregate:kspKotlin ?
c
I guess you could enforce that with 
mustRunAfter
or 
dependsOn
. If there's another mechanism to order KSP tasks, I'm not aware of it.
m
I expect the dependencies to do that. It’s cross modules so it uses the outgoing variants, not tasks
Copy code
// aggregate/build.gradle

// Depend on other module. They will be built before agggregate
dependencies {
  implementation(project(":module1"))
  implementation(project(":module2"))
}
the module1, module2 KSP task must run to produce the jar consumed by the aggregate module
c
That was my initial assumption as well. Things started getting weird when triggering a build to refresh a Jetpack Compose preview. At that point, it seems that the main module missed the first round of processing and went right to aggregation for some reason.
But sometimes it also works as expected, which made me think there was some kind of race happening.
And if I build and run the application, the same thing can happen 🤷
m
Might be an incremental compilation issue or something like that. If you can, upload a reproducer somewhere and file an issue
👍 1
c
Turning off incremental seems to have fixed it
z
Workaround would be a better word, turning off a critical feature isn’t really something I’d call a “fix” :)
c
Fair
I don’t know if what I was seeing was intended behavior or not. I also tried 
ksp.incremental.intermodule=true
, but it didn’t change anything with regard to the final output.
Would anyone be willing to take a look at this processor and let me know if it looks like it should work fine with incremental processing enabled? This sample project generates all expected files, so I’m guessing the issue I’m experiencing in the real project lies outside of KSP, but I want to make sure I’m not missing something fundamental first.
Nvm, I think I fixed it. I thought the default value for 
originatingKSFiles
in kotlin poet’s 
writeTo
was taking care of the source files for me somehow, but I see that it’s just aggregating from the `FileSpec`s members, and I still need to specify the files explicitly. 🤦‍♂️ Live and learn, I guess 🙂 Thanks for the help! This was a helpful hint: https://github.com/google/ksp/issues/1313
13 Views
Open in Slack
PreviousNext
Powered by

kotlinlang

A modern programming language that makes developers happier.

Powered by