My TestBalloon test suite is so fast the database ...

# testballoon

[CLOVIS]

My TestBalloon test suite is so fast the database is overloaded and becomes flaky 😅

Congrats, I did not expect unit tests to double as a benchmark

[Oliver.O]

I'm waiting for the day where folks come up with something like kotlinx-benchmark for TestBalloon.

[Bernd Prünster]

Such findings are exactly the kind of testing you get for free, when the test frameworks allows such loads to happen 😉

[CLOVIS]

I tried prototyping a kotlinx-benchmark thing for Prepared (thus would be compatible with TestBalloon) but the setup itself is not compatible with the idea of a benchmark. A proper benchmark should not be impacted by what programs are running, but between IntelliJ, Gradle, the Kotlin daemon, the JVM for the tests themselves… That's too much noise

[CLOVIS]

The JMH people don't recommend running benchmarks with JUnit for the same reason

[CLOVIS]

But yeah with the MongoDB thing I don't know what to do about it 😅 I could arbitrarily reduce the parallelism of my test suite, but that feels like a waste. My local machine can survive the test suite at full speed, the CI runner cannot

[Oliver.O]

Scale down parallelism on CI?

[CLOVIS]

Yeah, but I don't want to put an arbitrary low number. I wish it could auto-size itself, but I have no idea how that would be implemented

[Oliver.O]

That’s hard to get right. Often you don’t fully control the CI runners whose response time may vary as they service other people’s load due to virtualization.

A system should degrade gracefully under load but never fail. Any other behavior makes your life less enjoyable. kodee lost

[CLOVIS]

Yep.

Currently what happens:
• At some point, the DB becomes overloaded.
• It then takes 1–4 minutes to catch up, during which it stops communicating with the clients.
• Because the tests have a 1 minute timeout, the ones that were in progress at the moment appear as failed.
I've been tinkering with increasing the CPU priority of the DB, and that helps to a point, but that's just a band-aid, it just pushes the problem further.

I'm trying to figure out if I can detect that an overload is happening, and then maybe have the test suite slow itself somehow.

The other solution is just to disable parallelism, but that's obviously going to increase test times. Though I still prefer a slow test suite than a flaky one.

[Oliver.O]

Could you just massively increase the timeout?

[CLOVIS]

I can, but I'll need to set it to ~5 min, and that's very inconvenient if the database dies and it still runs for 20 or 30 minutes after that

[CLOVIS]

In the future I'll add Arrow Resilience to try and fail early, after that it's probably not that bad to have a large timeout

[Oliver.O]

I had even thought about setting it to like 60 minutes. But that implies that the timeout would practically never be the primary detector of a test failure. Multi-minute stalls are annoying if you don't have a heartbeat indicator in between.

[CLOVIS]

The CI runner kills itself after 1h anyway, so at that point the timeout is essentially disabled

[Bernd Prünster]

can you trim the number of test elements? that is our way of cheating resource-limited CI runners without having to hard-limit concurrency

[CLOVIS]

Do you mean having a single test that tests two things?

[CLOVIS]

The JVM that runs the tests themselves is fine, it's the DB instance that's struggling, I don't think that would change it

[CLOVIS]

(btw I set the timeout to 15 minutes and I'm seeing way fewer test failures, though now Gradle hangs forever after the tests are done for some reason… investigating)

[Bernd Prünster]

could be reporting that causes the hang. this is what i meant with reducing test elements. use virtualised ones and build an expressive test path from them, like we do

[Bernd Prünster]

it is the only way to work with test suites cuncurrently throwing millions of data points at your unit under test. if those are actual, real test elements that are supposed to show up as distinct entries in your test report, the receiving end of the report will glitch out, even if all tests succeeded

[CLOVIS]

Currently I have ~15k total tests, so it shouldn't be a problem yet? Also it's Gradle that's hanging, not the GitLab infra

[Bernd Prünster]

Yeah, that's pretty much nothing