Hi there this is kinda out of left field but something I ve

Question

Hi there this is kinda out of left field but something I ve been thinking about for a few days and kotest seems like my best bet I have a set of junit tests that end up spending a lot of time waiting for things using failsafe s retry Since kotest uses suspend functions and so do my test methods I m using for my e2e testing I was wondering about the feasibility of extending kotest to wrap each test case in a function that returns a deferred test result something like ```val tests List lt TestCase gt suspend fun executeAsync testCase TestCase Deferred lt TestResult gt fun doIt = tests map executeAsync it awaitAll or just start them instead``` If my failsafe retries were kotlin s `delay` then my other tests would run whatever they can instead of waiting for each test case to complete serially at least that s the way I understand it

Jim · Answer

The actual eventing to junit gradle could happen in serial as well I imagine since the result of `awaitAll` in my pseudocode would just be a list of `TestResult`s

sam · Answer

What about setting parallelism factor in kotest to run more specs in parallel

sam · Answer

It just creates a bigger thread pool atm but that would work if your failsafe tests block a thread

Jim · Answer

I tried that yesterday combining these ``` override val parallelism = 4 override val isolationMode = IsolationMode InstancePerTest``` but the tests within each spec even though they generate separate specs always ran in serial

sam · Answer

Yes that is the case atm You can set threads at the spec level too

Jim · Answer

sam · Answer

It could all be improved of course

Jim · Answer

ah let me try that

Jim · Answer

tbh I d prefer to use coroutines since failsafe is the only piece of non kotlin code i have in my suite but if I can get this working that would be the quick way forward

sam · Answer

I had an idea to allow people to set their own dispatcher for tests

Jim · Answer

If my idea isn t too insane and it s desirable I m also happy to do it

Jim · Answer

That would be cool

sam · Answer

In addition to a setting that allows all test cases in parallel and then you can achieve anything

sam · Answer

So each spec could be launched in a separate coroutine rather than a separate thread

Jim · Answer

That would be sweet for sure

sam · Answer

I don t think it would be that hard either

Jim · Answer

The other thought I had was to just have 1 spec per test case and move all of my before all junit nonsense into top level functions

Jim · Answer

but that s still blocked threads which is a bummer

sam · Answer

Before all will always need to complete before tests for obvious reasons

sam · Answer

Let me write up a ticket

Jim · Answer

pray thank you

sam · Answer

sam · Answer

please add feedback on that ticket

Jim · Answer

Looks perfect What kind of prioritization model do y all use Like I said I m happy to contribute on this if I can be pointed in the right direction to get the ball rolling at least thinking face

sam · Answer

We re pretty much ready to release 4 3 so this could fit into 4 4

Jim · Answer

thumbsup cool I can start retooling my work to use kotest then and then just take advantage of this when it arrives

sam · Answer

If you want to do this work feel free but I m happy to pick it up if you don t have time

Jim · Answer

Yeah I have a feeling that the spin up time for me would be a little too much and can afford to wait Implementing kotest will solve the defect I have on me and then when this MR goes through the runtime of my work will be much much better theoretically

sam · Answer

Yes in theory all you would need to do is flick the switch to active parallel execution of specs and then specify a dispatcher maybe by default it will use a single thread

sam · Answer

Even if you re available just to test from a snapshot once ready that would be great

Jim · Answer

Absolutely I started following the issue and can be pinged here thumbsup

sam · Answer

cool

Jim · Answer

I suspect `executeAndWait` is where some of the magic happens and that one looks pretty intense joy

sam · Answer

executeAndWait will use an executor per test case which will need to stay because we do that to detect deadlocked tests

sam · Answer

but I don t think that will matter

sam · Answer

just removing the spec executor thread pool will make a massive difference

Jim · Answer

Gotcha thumbsup

Jim · Answer

In the spirit of suspend what do we think about the callbacks in `SpecFunctionCallbacks` being marked with suspend

sam · Answer

can t really change those they have been around since 1 0

sam · Answer

well some of them have

Jim · Answer

fair enough

sam · Answer

if you want suspend versions you can use the inline ones

sam · Answer

or you can extend TestListener and those methods are suspend

Jim · Answer

inline thinking face

sam · Answer

so instead of overriding a function do this

sam · Answer

```class MyTest FunSpec init beforeSpec ```

sam · Answer

or ```class MyTest FunSpec beforeSpec ```

Jim · Answer

ahhh cool thanks smile

sam · Answer

all those ones are suspendable

Jim · Answer

eeeexcellent

Jim · Answer

I ve seen that init syntax in some places and not others is there a preferred method

sam · Answer

not really The lambda style is just a way of avoiding the init block I think the init block is ugly I prefer scala s way

sam · Answer

if you use the lambda style it just gets called in the parent spec s init method

Jim · Answer

thumbsup

Jim · Answer

```abstract class ValidationSpec body FunSpecBody = FunSpec body init beforeSpec assertServerAccessible afterSpec cleanupStuff ``` this is what i m going for

Jim · Answer

I ve got like 10 classes that all need this server and all produce junk of the same shape so I d like to write the cleanup once

Jim · Answer

I think the `body` param prevents me from doing it the scala way

sam · Answer

those are just functions so you can assign them as functions

sam · Answer

you can either do `beforeSpec setup ` where setup is a function `Spec gt Unit` or whatever the spec is

Jim · Answer

ah gotcha thumbsup

sam · Answer

Or you can do `object class Something TestListener override here ` and pass it into multiple specs via `listener mything `

sam · Answer

that second approach can be at the global level too via project config or Autoscan

sam · Answer

loads of possbilities

Jim · Answer

smiling imp

Jim · Answer

the only thing left for me to figure out is how to avoid `lateinit var` e g ```lateinit var version Version beforeSpec version = getServerVersion ``` but perhaps just assigning version and allowing it to happen at construction of the spec time is fine since it really needs to happen before any tests not necessarily before the spec thinking face

sam · Answer

if you can create Version when you create the listener you can do something like ```object MyListener TestListener val version Version = override fun beforeSpec override fun afterSpec ```

sam · Answer

then in your test ```class MyTest FunSpec init listener MyListener test xx ```

Jim · Answer

but then version wouldn t be accessible to the test cases right

sam · Answer

but if you can t create the instance until beforeSpec you will need a var or lateinit somewhere

sam · Answer

```class MyTest FunSpec init val listener = listener MyInstance test xx use listener here ```

sam · Answer

If its an object then it doesn t matter just refer to the object directly

Jim · Answer

oh of course thumbsup

sam · Answer

```val processorListener = IngestionProcessorTestListener listener startStopKafka listener processorListener listener resetDatabase ```

sam · Answer

I do things like this in my tests

sam · Answer

```val startStopKafka = object TestListener override suspend fun beforeSpec spec Spec EmbeddedKafka start EmbeddedKafkaConfig defaultConfig while EmbeddedKafka isRunning Thread sleep 100 override suspend fun afterSpec spec Spec EmbeddedKafka stop ```

Jim · Answer

and the order of the afterSpec functions are run in order the listeners are registered

sam · Answer

I think technically we make no guarantees

sam · Answer

because you can register them in about 200 places

Jim · Answer

gotcha

sam · Answer

I think for 4 4 we should make it explicit the order

sam · Answer

but right now it will be registration order yes

sam · Answer

but don t hold us to it wink

sam · Answer

if you want to be sure just make a composite listener

sam · Answer

then your composite listener can call them in order

Jim · Answer

insert mind blown emoji here

Jim · Answer

this is far and away more advanced than junit joy I knew this was the right call

sam · Answer

yeah it s using a proper functional programming approach where possible there s vars under the hood and magic dsl at play too

sam · Answer

so it s way way more powerful than shoving annotations on a few things

sam · Answer

And if you need to go to 11 you can use extensions which give you full control over the engine so you can do around advice You can override test results decide if tests get invoked and so on

Jim · Answer

smiling imp

Jim · Answer

for now Tags and EnabledIf are really great for what I need

sam · Answer

eg <https github com kotest kotest blob master kotest framework kotest framework api src commonMain kotlin io kotest core extensions TestCaseExtension kt>

sam · Answer

Yes enabled if tags listeners are more than powerful enough for 99 of cases and then the extensions for when you need to go to 11

Jim · Answer

so I could implement the MR at the top with the TestCaseExtension run my suspend funs in mostly parallel

sam · Answer

the test case extension is good if you want to do something like ```code here runTest code here```

Jim · Answer

gotcha so I m still at the mercy of the engine currently running each test case in order and serially until the feature is added

sam · Answer

they re more for doing plugins and stuff but you can use them for whatever

sam · Answer

until the feature we discussed other day yes

sam · Answer

in the meantime just set parallelism to like 10 and use threads for it

sam · Answer

it s not coroutines and a big hammer but would do a job for now

Jim · Answer

right thumbsup

Jim · Answer

but first rip out junit everywhere joy

sam · Answer

yep

sam · Answer

that s a fun job though I quite enjoy things like that

Jim · Answer

same here it s nice seeing how much better you can make things when you use better tools