Prepared: TestBalloon's API is stabilizing, which ...

# opensavvy

[Oliver.O]

Prepared: TestBalloon's API is stabilizing, which also applies to the annotations guiding the IDE plugin. @TestElementName, which is currently not used by the IDE plugin, will become relevant. And the intention is to drop the IDE plugin's configurable names at some point. Is there a migration plan on the Prepared side? Anything special to consider on the TestBalloon side?

[CLOVIS]

Prepared doesn't depend on the TestBalloon annotations, instead it declares its own @TestEntrypoint. So, no worries of binary compatibility etc.

The :compat-testballoon module does use the TestBalloon annotations, but the stability guarantees for that module is that it has the same stability as TestBalloon itself, so whatever your deprecation cycle is fine there.

So the only real change required will be in the documentation to explain how to use the new version of the TestBalloon plugin, but that can be a few days out of sync without breaking anything.

[Oliver.O]

That sounds cool! I was a little worried because of @TestEntrypoint. There is no way to expect/actual annotations, or is there?

[CLOVIS]

I think you can

[CLOVIS]

But that wouldn't be usable for me because I don't use expect/actual, it's different implementations of the same interface.

[Oliver.O]

Maybe it's actually not relevant here. Just occurred to me that it might have been a solution when we were talking about annotation name stability a while ago.

[CLOVIS]

To summarize: as long as the IntelliJ plugin continues to have some way to mark that opensavvy.prepared.suite.annotations.TestEntrypoint marks entrypoints, I don't think there's anything special required

[CLOVIS]

And apparently I've never marked that annotation as stable, so actually I could also change it if needed

[Oliver.O]

Yes, it's this kind of configuration that will cease to exist. So the IDE plugin will rely on TestBalloon's annotations where necessary and not iterate over multiple (configurable) annotations. Stability on that side is a bit illusionary anyway as the actual static analysis mechanism is quite complex and binary compat will never cover subtle changes in semantics.

[CLOVIS]

You mentioned @TestElementName will have a new meaning, what will the resolution do with it?

[Oliver.O]

Not a new meaning, but being used by the IDE plugin at all. Until now, it was only used as a hint for the compiler plugin.

What will also change:

The @TestElementName annotation now supports optional prefix and postfix parameters, making the IDE plugin aware of modifications to the test element name.

Could it work if you'd have expect TestEntrypoint in common code and actual that to TestRegistering?

[CLOVIS]

I can't expect-actual because the DSL is defined in :suite but only :compat-testballoon knows about TestBalloon

[Oliver.O]

Ah, I see. We'd need cross-module expect-actual for that.

[CLOVIS]

I guess the two main solutions:
• You publish a testballoon-annotations module that you swear provides binary stability. You can still deprecate stuff or create v2 annotations, just not binary-break the existing ones
• Or, I create matching annotations on my side and the IntelliJ plugin knows that they are equivalents to the ones provided by TestBalloon itself?

[CLOVIS]

It's not really a problem if the annotations stop being used in the future, or if they change meaning, it won't break any code. The only issue is if they have a binary breaking change, since I promise I won't have those until a major version

[Oliver.O]

testBalloon-framework-shared is probably already stable enough for that purpose, so that could be used. And the IDE plugin will always recognize annotations by FQN, so mirroring will work, but collide on the classpath/linkage side, right?

Once TestBalloon is at 1.0 (which I guess is not too far away), the usual binary guarantees apply, so only major releases are expected to break things.

[CLOVIS]

> And the IDE plugin will always recognize annotations by FQN, so mirroring will work, but collide on the classpath/linkage side, right?
If I create the exact same annotations in my package, and you tweak the IDE plugin to recognize both the one in the TestBalloon package and the one in the Prepared package, there's no worries about binary compatibility or syncing us both to do breaking change in the future

[CLOVIS]

It probably won't add much code in the plugin since the idea is that they both do the exact same thing

[Oliver.O]

So different FQNs. That would be what we have now, just non-configurable. And the code is there, it‘s just about the long-term implications of keeping it.

[CLOVIS]

What do you think?

I don't plan on making any changes to the annotation. But also if you want to make changes to yours, you can do it without fear of breaking anything.

[Oliver.O]

I guess in the end it’s more than just annotations. Any change inside the IDE plugin could introduce some incompatibility.

[CLOVIS]

What kind of incompatibilities?

My understanding is the IDE patterns-match on the code to find the annotations. If it does, it enables additional behavior.

I think that worst-case, the IDE changes logic and doesn't recognize Prepared's variant of the annotations until they're updated, and for some time the IDE doesn't add the green arrow in front of Prepared methods anymore. If so, that's acceptable if it's rare, the projects are not broken and it's still possible to run all tests per module via Gradle. It's just temporarily not possible to run tests individually, but it's already not possible to do that when using Prepared with kotlin-test or Kotest, so…

[CLOVIS]

Unless there's a bigger problem?

[Oliver.O]

There is no pattern matching, but rather a combination of using the IDE‘s syntax tree (PSI) plus full-blown Kotlin compiler analysis. And since it happens in an editor, changes to the source happen all the time, and the plugin needs to react to callbacks asking for an analysis anywhere in the code (so not just top-down parsing). I don’t have a complete picture of possible failure modes, but my understanding is always biased toward what I know about TestBalloon‘s structure.

So if you were just building on the TestBalloon public API, that would be safe. Anything else is more or less a lucky coincidence. But yes, all that can break is gutter icons missing or appearing at the wrong places, and run configurations from gutter icons not working properly.

[CLOVIS]

I can't build on top of the TestBalloon public API though, because it's not stable. Even if it was, the goal of Prepared is to work identically no matter which framework you're using (yes, I'm still hoping we one day get test framework primitives in the stdlib). If :suite had to depend on all frameworks that are possibly used, users would need to have dependencies on Kotest, kotlin-test and even the KTI prototype. That's not a future-proof approach.

That leaves the other option then; extract the annotations in :testballoon-annotations or similar. You can keep them in the same package, this will be completely invisible to your users. But other libraries can make themselves visible by the TestBalloon plugin, without needing to have a full dependency on TestBalloon itself, and all the compatibility work that would require.

[CLOVIS]

This is already what JetBrains themselves do with jetbrains-annotations , what KtMongo does, etc

[Oliver.O]

We still have some time until something materializes, but as I said: A mostly stable package with annotations is already there. But as I mentioned, the entire scheme comprises more than just annotation stability. Once you get into the details, I doubt that there will ever be a common test framework abstraction that covers all the cases we already know and is flexible enough to allow future progress.