Hi all :wave: I spent abit of time this weekend to...
# testballoon
y
Hi all 👋 I spent abit of time this weekend to play with the library especially around SpringBoot and came up with the wrapper https://github.com/yonatankarp/testballoon-spring - I think though that at least the speed gain in term of performance won't be much for spring since it does most of the discovery work with reflection sad panda
o
Thanks for sharing your project and experiences, super interesting! Which speed gains in particular have you been looking for?
y
I wasn't specifically looking for performance improvements. After your presentation at KUG Berlin last week, I saw that testBalloon performs better than JUnit/Kotest in many cases, which is why I mentioned it. I just meant that Spring still does a lot of reflection and context initialization, so I'd expect the performance gains to be smaller there. 🙂
o
Just to clarify: The performance gains I was showing resulted from massive parallelization, which TestBalloon can do to the highest degree available. I also wanted to have a real-world benchmark, not some artificial one which could be biased or irrelevant (e.g. the distribution of long vs. short running tests can make a difference). So I took that actual migration project and compared Kotest with TestBalloon, where sequential execution was at the same level, but parallel execution showed huge differences. I would have loved to include JUnit in such a comparison, but there was no real-world use case for that (unfortunately, many test setups are closed source, especially the larger ones). Now about your case: No test framework can improve performance inside tests. With sequential execution, the only gains you could expect is the framework's overhead: JUnit uses reflection extensively for test discovery, and if there is a very large number of tests that may be significantly slower than Kotest or TestBalloon (both use compile-time discovery). But if your tests are isolated (do not share unsynchronized global mutable state) and allow parallelization, the framework's scheduling matters and at scale, TestBalloon will shine. I don't know if that could be done with Spring: Does it rely on shared unsynchronized global mutable state? Can we create isolated tests which could run in parallel without mutual interference?
y
Thanks, that matches what I expected, and it's a fair way to benchmark. No disagreement on the framing: nothing speeds up the work inside a test, a sequential run only recovers the framework's own overhead, and the real gains come from scheduling and isolation, which need scale to show. Compile-time discovery is a genuine win over JUnit's reflection, but on a Spring suite it's small next to context bootstrap, so I wouldn't expect it to matter on its own. On whether Spring relies on shared unsynchronized global mutable state: partly, but it's bounded and nameable rather than a vague hazard. Spring singletons are built to be thread-safe under production concurrency, so the
ApplicationContext
itself isn't what breaks parallel tests. What serializes a suite is the database (and anything external the beans touch), plus the JVM-wide
TestContext
cache that shares singletons across test classes with identical config. In my wrapper there's one more piece: inside a single
springTest
block the tests share one context and a set of mocks that get `clearMocks`'d before each test, so those mocks are shared mutable state too. On building isolated tests that parallelize: yes, at the granularity of a suite rather than an individual test, and the wrapper already works that way. Each
springTest
block builds its own context, and the registry that hands the mocks to it is a
ThreadLocal
, so two suites loading on different threads don't interfere. What isn't parallel-safe is the tests inside one block: they share the context and the mocks, and one of my own suites even depends on ordering and a shared
var
. So the unit of parallelism here is the suite, internally sequential, which is the level your scheduler would work at. I guess that the main concern is that SpringBoot isn't Kotlin first framework and while supporting Coroutines OOTB they're not 1st class citizens (for example the
ThreadLocal
above)
o
Everything you wrote makes perfect sense to me. I'd still have to look into the details (and I will).
On building isolated tests that parallelize: yes, at the granularity of a suite rather than an individual test
This can be expressed with TestBalloon, as we can switch between sequential and concurrent invocation at any level of the test element hierarchy. For example, we could use concurrent invocation for the entire session
Copy code
TestSession(testConfig = TestConfig.invocation(TestConfig.Invocation.Concurrent))
and then choose
Copy code
testConfig = TestConfig.invocation(TestConfig.Invocation.Sequential)
specifically for the respective Spring Boot test suites.
I guess that the main concern is that SpringBoot isn't Kotlin first framework and while supporting Coroutines OOTB they're not 1st class citizens (for example the
ThreadLocal
above)
I probably need to dig into this a bit further, but having a system under test which assumes a threading model doesn't necessarily hurt. Coroutines can provide threading guarantees if needed (e.g. UI-based systems typically rely on a main thread). And even if not used inside tests, coroutines can help organize parallel execution. But if I understand your library correctly, like elsewhere with TestBalloon, we can immediately use plain Kotlin to parameterize and structure tests. Is this something you'd like to maintain as an external integration for TestBalloon? Or would you prefer having Spring Boot support inside TestBalloon? In any case, I'd be happy to look into all the details and discuss implementation aspects.
d
Funny, I was just trying out something similar with Micronaut tests... I think they have the same problems (initializing an ApplicationContext per test, and threading), although they rely on reflection much less. At first, the tests ran much slower, then I tried one ApplicationContext per suite, which reduced the per test overhead, but the startup was still a bit long (jvm init?)... that particular microservice doesn't have tons of tests yet, so I can't yet speak about other considerations though...
y
@Oliver.O thanks for the tips! I wasn't aware of that! As for the library - it can probably be a module in the library itself, the main reason I created it was that i needed a bridge for my pet projects 🙂 Also, there are some limitations i couldn't find work around for that are annotation based spring specific like
@DirtyCotnext
and
@Tranaactional
annotation on a method for example
o
Thanks for trying this far! I'll try to look into all of this and see if I can find out how to address the remaining gaps you mentioned. But it will take some time, as I'd have to become more familiar with Spring.
💯 1