https://kotlinlang.org logo
Title
j

Jim

10/07/2020, 5:20 PM
Hi there, this is kinda out of left field but something I've been thinking about for a few days and kotest seems like my best bet. I have a set of junit tests that end up spending a lot of time waiting for things using failsafe's retry. Since kotest uses suspend functions (and so do my test methods I'm using for my e2e testing) I was wondering about the feasibility of extending kotest to wrap each test case in a function that returns a deferred test result, something like:
val tests: List<TestCase>
suspend fun executeAsync(testCase: TestCase...): Deferred<TestResult>...
fun doIt() = tests.map { executeAsync(it) }.awaitAll() // or just start them instead
If my failsafe retries were kotlin's
delay
then my other tests would run whatever they can instead of waiting for each test case to complete serially, at least that's the way I understand it.
The actual eventing to junit/gradle could happen in serial as well I imagine since the result of
awaitAll
in my pseudocode would just be a list of `TestResult`s
s

sam

10/07/2020, 5:22 PM
What about setting parallelism factor in kotest to run more specs in parallel
It just creates a bigger thread pool atm but that would work if your failsafe tests block a thread
j

Jim

10/07/2020, 5:25 PM
I tried that yesterday combining these:
override val parallelism = 4
    override val isolationMode = IsolationMode.InstancePerTest
but the tests within each spec even though they generate separate specs always ran in serial
s

sam

10/07/2020, 5:25 PM
Yes that is the case atm. You can set threads at the spec level too.
s

sam

10/07/2020, 5:25 PM
It could all be improved of course
j

Jim

10/07/2020, 5:25 PM
ah, let me try that
tbh I'd prefer to use coroutines since failsafe is the only piece of non kotlin code i have in my suite, but if I can get this working that would be the quick way forward
s

sam

10/07/2020, 5:26 PM
I had an idea to allow people to set their own dispatcher for tests
j

Jim

10/07/2020, 5:26 PM
If my idea isn't too insane and it's desirable I'm also happy to do it
That would be cool
s

sam

10/07/2020, 5:27 PM
In addition to a setting that allows all test cases in parallel, and then you can achieve anything
So each spec could be launched in a separate coroutine rather than a separate thread
j

Jim

10/07/2020, 5:29 PM
That would be sweet for sure
s

sam

10/07/2020, 5:29 PM
I don't think it would be that hard either
j

Jim

10/07/2020, 5:29 PM
The other thought I had was to just have 1 spec per test case and move all of my "before all" junit nonsense into top level functions
but that's still blocked threads which is a bummer
s

sam

10/07/2020, 5:30 PM
Before all will always need to complete before tests for obvious reasons
Let me write up a ticket
j

Jim

10/07/2020, 5:34 PM
๐Ÿ™ thank you!
please add feedback on that ticket
๐Ÿ‘ 1
j

Jim

10/07/2020, 5:59 PM
Looks perfect! What kind of prioritization model do y'all use? Like I said I'm happy to contribute on this if I can be pointed in the right direction to get the ball rolling at least ๐Ÿค”
s

sam

10/07/2020, 5:59 PM
We're pretty much ready to release 4.3, so this could fit into 4.4
j

Jim

10/07/2020, 6:00 PM
๐Ÿ‘ cool, I can start retooling my work to use kotest then and then just take advantage of this when it arrives
s

sam

10/07/2020, 6:01 PM
If you want to do this work, feel free, but I'm happy to pick it up if you don't have time.
j

Jim

10/07/2020, 6:03 PM
Yeah I have a feeling that the spin up time for me would be a little too much and can afford to wait. Implementing kotest will solve the defect I have on me, and then when this MR goes through the runtime of my work will be much much better (theoretically)
s

sam

10/07/2020, 6:04 PM
Yes in theory, all you would need to do is flick the switch to active parallel execution of specs, and then specify a dispatcher (maybe by default it will use a single thread)
Even if you're available just to test from a snapshot once ready, that would be great.
j

Jim

10/07/2020, 6:14 PM
Absolutely, I started following the issue, and can be pinged here ๐Ÿ‘
s

sam

10/07/2020, 6:14 PM
cool
j

Jim

10/07/2020, 6:41 PM
I suspect
executeAndWait
is where some of the magic happens and that one looks pretty intense ๐Ÿ˜‚
s

sam

10/07/2020, 6:41 PM
executeAndWait will use an executor per test case, which will need to stay because we do that to detect deadlocked tests
but I don't think that will matter
just removing the spec executor thread pool will make a massive difference
j

Jim

10/07/2020, 6:42 PM
Gotcha ๐Ÿ‘
In the spirit of suspend, what do we think about the callbacks in
SpecFunctionCallbacks
being marked with suspend?
s

sam

10/09/2020, 5:54 PM
can't really change those, they have been around since 1.0
well some of them have
j

Jim

10/09/2020, 5:54 PM
fair enough
s

sam

10/09/2020, 5:54 PM
if you want suspend versions you can use the inline ones
or you can extend TestListener and those methods are suspend
j

Jim

10/09/2020, 5:55 PM
inline ๐Ÿค”
s

sam

10/09/2020, 5:55 PM
so instead of overriding a function, do this
class MyTest : FunSpec() {
  init {
     beforeSpec { ... }
   }
}
or
class MyTest : FunSpec({
  beforeSpec { ... }
})
j

Jim

10/09/2020, 5:56 PM
ahhh cool! thanks ๐Ÿ˜„
s

sam

10/09/2020, 5:56 PM
all those ones are suspendable
j

Jim

10/09/2020, 5:56 PM
eeeexcellent
I've seen that init syntax in some places and not others, is there a preferred method?
s

sam

10/09/2020, 5:59 PM
not really. The lambda style is just a way of avoiding the init block. I think the init block is ugly, I prefer scala's way.
if you use the lambda style, it just gets called in the parent spec's init method
j

Jim

10/09/2020, 5:59 PM
๐Ÿ‘
abstract class ValidationSpec(body: FunSpecBody = {}) : FunSpec(body) {
    init {
        beforeSpec { 
            assertServerAccessible()
        }
        
        afterSpec { 
            cleanupStuff()
        }
    }
this is what i'm going for
I've got like 10 "classes" that all need this server, and all produce junk of the same shape so I'd like to write the cleanup once
I think the
body
param prevents me from doing it the scala way
s

sam

10/09/2020, 6:26 PM
those are just functions, so you can assign them as functions
you can either do
beforeSpec(setup)
where setup is a function
Spec -> Unit
or whatever the spec is
j

Jim

10/09/2020, 6:27 PM
ah gotcha ๐Ÿ‘
s

sam

10/09/2020, 6:27 PM
Or you can do
object/class Something : TestListener { ... override here }
and pass it into multiple specs via
listener(mything)
that second approach can be at the global level too via project config or @Autoscan
loads of possbilities
j

Jim

10/09/2020, 6:27 PM
๐Ÿ˜ˆ
the only thing left for me to figure out is how to avoid
lateinit var
e.g.:
lateinit var version: Version

beforeSpec {
  version = getServerVersion()
}
but perhaps just assigning version and allowing it to happen at construction of the spec time is fine, since it really needs to happen before any tests not necessarily before the spec ๐Ÿค”
s

sam

10/09/2020, 6:35 PM
if you can create Version when you create the listener, you can do something like
object MyListener: TestListener { 
  val version: Version = ....
  override fun beforeSpec ...
  override fun afterSpec...
}
then in your test
class MyTest:  FunSpec {
  init {
     listener(MyListener)
      test("xx") { }
   }
}
j

Jim

10/09/2020, 6:35 PM
but then version wouldn't be accessible to the test cases right?
s

sam

10/09/2020, 6:35 PM
but if you can't create the instance until beforeSpec you will need a var or lateinit somewhere
class MyTest:  FunSpec {
  init {
      val listener = listener(MyInstance())
      test("xx") { 
        // use listener here
      }
   }
}
If its an object then it doesn't matter, just refer to the object directly
j

Jim

10/09/2020, 6:36 PM
oh of course ๐Ÿ‘
s

sam

10/09/2020, 6:39 PM
val processorListener = IngestionProcessorTestListener()

listener(startStopKafka)
listener(processorListener)
listener(resetDatabase)
I do things like this in my tests
val startStopKafka = object : TestListener {

   override suspend fun beforeSpec(spec: Spec) {
      EmbeddedKafka.start(EmbeddedKafkaConfig.defaultConfig())
      while (!EmbeddedKafka.isRunning()) {
         Thread.sleep(100)
      }
   }

   override suspend fun afterSpec(spec: Spec) {
      EmbeddedKafka.stop()
   }
}
j

Jim

10/09/2020, 6:40 PM
and the order of the afterSpec functions are run in order the listeners are registered?
s

sam

10/09/2020, 6:40 PM
I think technically we make no guarantees
because you can register them in about 200 places
j

Jim

10/09/2020, 6:41 PM
gotcha
s

sam

10/09/2020, 6:41 PM
I think for 4.4 we should make it explicit the order
but right now, it will be registration order yes
but don't hold us to it ๐Ÿ˜‰
if you want to be sure, just make a composite listener
then your composite listener can call them in order
j

Jim

10/09/2020, 6:42 PM
insert mind-blown emoji here
this is far and away more advanced than junit ๐Ÿ˜‚ I knew this was the right call
s

sam

10/09/2020, 6:47 PM
yeah it's using a proper functional programming approach (where possible, there's vars under the hood, and magic dsl at play too)
so it's way way more powerful than shoving annotations on a few things
And if you need to go to 11, you can use extensions, which give you full control over the engine, so you can do around advice. You can override test results, decide if tests get invoked and so on
j

Jim

10/09/2020, 6:48 PM
๐Ÿ˜ˆ
for now, Tags and EnabledIf are really great for what I need
Yes enabled if, tags, listeners, are more than powerful enough for 99% of cases, and then the extensions for when you need to go to 11.
j

Jim

10/09/2020, 6:49 PM
so I could implement the MR at the top with the TestCaseExtension? (run my suspend funs in "mostly parallel")
s

sam

10/09/2020, 6:49 PM
the test case extension is good if you want to do something like
code here
runTest()
code here
j

Jim

10/09/2020, 6:50 PM
gotcha, so I'm still at the mercy of the engine currently running each test case in order and serially until the feature is added
s

sam

10/09/2020, 6:50 PM
they're more for doing plugins and stuff, but you can use them for whatever
until the feature we discussed other day yes
in the meantime just set parallelism to like 10 and use threads for it
it's not coroutines and a big hammer, but would do a job for now
j

Jim

10/09/2020, 6:51 PM
right ๐Ÿ‘
but first, rip out junit everywhere ๐Ÿ˜‚
s

sam

10/09/2020, 6:51 PM
yep
that's a fun job though, I quite enjoy things like that
j

Jim

10/09/2020, 6:51 PM
same here, it's nice seeing how much better you can make things when you use better tools
๐Ÿ‘๐Ÿป 1