For Anvil’s KSP support, I’m trying to figure out ...

# ksp

[Zac Sweers]

For Anvil’s KSP support, I’m trying to figure out how to strike the right balance with incremental support. The way anvil works is it writes “hints” to known packages and then fishes them out during component merging later. While KSP doesn’t offer a way to get arbitrary packages, it does offer a mechanism to get all files. However, this only includes the current compilation unit and now the whole classpath. So, currently I’m working around it by pulling out the underlying module and reading classpath data from that.

For incremental support, we don’t have access to KSFile instances to specify. Does it matter if they’re not in the compilation unit? Or would we need to consider marking all files as inputs to be really safe?

[Ting-Yuan Huang]

Do you mean what happens if there's a change to a (direct or indirect) dependency of the current compilation unit? In a KSP processor, symbols / elements in classpath can be obtained in two ways:

1. (root) Obtained directly via`Resolver.getClassDeclarationByName`: In this case, they should be associated to the output file(s) explicitly via CodeGenerator.associateWithClasses
2. (non-root) Obtained indirectly via resolving a type reference. For example, the KSClassDeclaration of Int can be obtained by resolving the type of val x: Int and then KSType.declaration. The root of this reference / navigation chain should be associated with the corresponding outputs.
When there is a change in the classpath, KSP invalidates the navigation / visit / reference chains all the way toward roots. Source files that are reachable from the changes are considered dirty and will get re-processed.

In other words, whenever there is a change in the compile classpath / dependencies, KSP will invoke the processors and pass them source files that are considered dirty, even when none of the source files are dirty. Note that `Resolver.getAllFiles`/ Resolver.getSymbolsWithAnnotation only returns / considers dirty files.

[Zac Sweers]

Apologies for the delay on getting back to you!

I think the missing link for us is that while we can associate classes accordingly, we’d also need to trigger getSymbolsWithAnnotation() to show up if new changes are present in that known package. Do you know if there’s a way we could implicitly consider all symbols within a package as roots? Essentially I’m thinking CodeGenerator.associateWithPackage(). Dagger/Hilt does something similar with a shared package, but works around this by inspecting the classpath prior to compilation and generating a processable source file that KSP can use instead.

[Ting-Yuan Huang]

To be clear, CodeGenerator.associateWithClass(outFile1, ...) does not cause a class in classpath / library to appear in getSymbolsWithAnnotation. It only causes the sources that are associated with outFile1 in the current compilation unit to be reprocessed.

If, with CodeGenerator.associateWithPackage(outFile1, pkg1), you mean that changes to or within pkg1 should cause sources that are associated with outFile1 in the currently compilation unit to be considered / returned by getSymbolsWithAnntoation, then declaring the outFile1 to be aggregating might be what you need; Any changes in classpath will invalidate all aggregating outputs, and therefore their input files. Of course, that can be somewhat less efficient than a finer grained package-only API, but if it is already aggregating, there won't be any difference.

If you mean a way to return elements in libraries by getSymbolsWithAnnotation, I'm afraid that could be slow. KSP would have to scan the classpath unconditionally and it can do no faster than other approaches. Similar to Anvil, you might write some hints to the annotation of a predefined class in each module, and reference + read them in the current compilation unit instead.

We do have Resolver.getDeclarationsFromPackage and scanning with it should be easier than scanning jars. It isn't immune to the aforementioned performance implications though, so please use it at your discretion. and don't forget to declare the output aggregating.

[Zac Sweers]

I think most of the pieces are there, but my main concern would be a scenario where a new hint is added to that known package and isn’t seen as an associated file that needs to retrigger processing

[Zac Sweers]

Slower is fine in think in this case since it would be opt-in

[Ting-Yuan Huang]

Marking the output aggregating should address your concern. On the performance side, if your hints have their own dedicated package, e.g., hint files are in the form com.hint.<deterministic-sparse-hash> instead of hiding in all packages, then scanning a relatively small com.hint should be ok.

[Zac Sweers]

Yeah they’re all in the same package

[Zac Sweers]

I didn’t realize marking it as aggregating would always invalidate roots 🤔. I assumed it would only invalidate if files affected in the same compilation unit changed

[Ting-Yuan Huang]

Honestly I hesitated when doing that. Now it looks like this somewhat conservative decision is better.