https://kotlinlang.org logo
Join Slack
Powered by
Hi there, this is kinda out of left field but some...
# kotest
j
Hi there, this is kinda out of left field but something I've been thinking about for a few days and kotest seems like my best bet. I have a set of junit tests that end up spending a lot of time waiting for things using failsafe's retry. Since kotest uses suspend functions (and so do my test methods I'm using for my e2e testing) I was wondering about the feasibility of extending kotest to wrap each test case in a function that returns a deferred test result, something like:
Copy code
val tests: List<TestCase>
suspend fun executeAsync(testCase: TestCase...): Deferred<TestResult>...
fun doIt() = tests.map { executeAsync(it) }.awaitAll() // or just start them instead
If my failsafe retries were kotlin's 
delay
then my other tests would run whatever they can instead of waiting for each test case to complete serially, at least that's the way I understand it.
The actual eventing to junit/gradle could happen in serial as well I imagine since the result of 
awaitAll
in my pseudocode would just be a list of `TestResult`s
s
What about setting parallelism factor in kotest to run more specs in parallel
It just creates a bigger thread pool atm but that would work if your failsafe tests block a thread
j
I tried that yesterday combining these:
Copy code
override val parallelism = 4
    override val isolationMode = IsolationMode.InstancePerTest
but the tests within each spec even though they generate separate specs always ran in serial
s
Yes that is the case atm. You can set threads at the spec level too.
j
s
It could all be improved of course
j
ah, let me try that
tbh I'd prefer to use coroutines since failsafe is the only piece of non kotlin code i have in my suite, but if I can get this working that would be the quick way forward
s
I had an idea to allow people to set their own dispatcher for tests
j
If my idea isn't too insane and it's desirable I'm also happy to do it
That would be cool
s
In addition to a setting that allows all test cases in parallel, and then you can achieve anything
So each spec could be launched in a separate coroutine rather than a separate thread
j
That would be sweet for sure
s
I don't think it would be that hard either
j
The other thought I had was to just have 1 spec per test case and move all of my "before all" junit nonsense into top level functions
but that's still blocked threads which is a bummer
s
Before all will always need to complete before tests for obvious reasons
Let me write up a ticket
j
🙏 thank you!
s
please add feedback on that ticket
👍 1
j
Looks perfect! What kind of prioritization model do y'all use? Like I said I'm happy to contribute on this if I can be pointed in the right direction to get the ball rolling at least 🤔
s
We're pretty much ready to release 4.3, so this could fit into 4.4
j
👍 cool, I can start retooling my work to use kotest then and then just take advantage of this when it arrives
s
If you want to do this work, feel free, but I'm happy to pick it up if you don't have time.
j
Yeah I have a feeling that the spin up time for me would be a little too much and can afford to wait. Implementing kotest will solve the defect I have on me, and then when this MR goes through the runtime of my work will be much much better (theoretically)
s
Yes in theory, all you would need to do is flick the switch to active parallel execution of specs, and then specify a dispatcher (maybe by default it will use a single thread)
Even if you're available just to test from a snapshot once ready, that would be great.
j
Absolutely, I started following the issue, and can be pinged here 👍
s
cool
j
I suspect 
executeAndWait
is where some of the magic happens and that one looks pretty intense 😂
s
executeAndWait will use an executor per test case, which will need to stay because we do that to detect deadlocked tests
but I don't think that will matter
just removing the spec executor thread pool will make a massive difference
j
Gotcha 👍
In the spirit of suspend, what do we think about the callbacks in 
SpecFunctionCallbacks
being marked with suspend?
s
can't really change those, they have been around since 1.0
well some of them have
j
fair enough
s
if you want suspend versions you can use the inline ones
or you can extend TestListener and those methods are suspend
j
inline 🤔
s
so instead of overriding a function, do this
Copy code
class MyTest : FunSpec() {
  init {
     beforeSpec { ... }
   }
}
or
Copy code
class MyTest : FunSpec({
  beforeSpec { ... }
})
j
ahhh cool! thanks 😄
s
all those ones are suspendable
j
eeeexcellent
I've seen that init syntax in some places and not others, is there a preferred method?
s
not really. The lambda style is just a way of avoiding the init block. I think the init block is ugly, I prefer scala's way.
if you use the lambda style, it just gets called in the parent spec's init method
j
👍
Copy code
abstract class ValidationSpec(body: FunSpecBody = {}) : FunSpec(body) {
    init {
        beforeSpec { 
            assertServerAccessible()
        }
        
        afterSpec { 
            cleanupStuff()
        }
    }
this is what i'm going for
I've got like 10 "classes" that all need this server, and all produce junk of the same shape so I'd like to write the cleanup once
I think the 
body
param prevents me from doing it the scala way
s
those are just functions, so you can assign them as functions
you can either do 
beforeSpec(setup)
where setup is a function 
Spec -> Unit
or whatever the spec is
j
ah gotcha 👍
s
Or you can do 
object/class Something : TestListener { ... override here }
and pass it into multiple specs via 
listener(mything)
that second approach can be at the global level too via project config or @Autoscan
loads of possbilities
j
😈
the only thing left for me to figure out is how to avoid 
lateinit var
e.g.:
Copy code
lateinit var version: Version

beforeSpec {
  version = getServerVersion()
}
but perhaps just assigning version and allowing it to happen at construction of the spec time is fine, since it really needs to happen before any tests not necessarily before the spec 🤔
s
if you can create Version when you create the listener, you can do something like
Copy code
object MyListener: TestListener { 
  val version: Version = ....
  override fun beforeSpec ...
  override fun afterSpec...
}
then in your test
Copy code
class MyTest:  FunSpec {
  init {
     listener(MyListener)
      test("xx") { }
   }
}
j
but then version wouldn't be accessible to the test cases right?
s
but if you can't create the instance until beforeSpec you will need a var or lateinit somewhere
Copy code
class MyTest:  FunSpec {
  init {
      val listener = listener(MyInstance())
      test("xx") { 
        // use listener here
      }
   }
}
If its an object then it doesn't matter, just refer to the object directly
j
oh of course 👍
s
Copy code
val processorListener = IngestionProcessorTestListener()

listener(startStopKafka)
listener(processorListener)
listener(resetDatabase)
I do things like this in my tests
Copy code
val startStopKafka = object : TestListener {

   override suspend fun beforeSpec(spec: Spec) {
      EmbeddedKafka.start(EmbeddedKafkaConfig.defaultConfig())
      while (!EmbeddedKafka.isRunning()) {
         Thread.sleep(100)
      }
   }

   override suspend fun afterSpec(spec: Spec) {
      EmbeddedKafka.stop()
   }
}
j
and the order of the afterSpec functions are run in order the listeners are registered?
s
I think technically we make no guarantees
because you can register them in about 200 places
j
gotcha
s
I think for 4.4 we should make it explicit the order
but right now, it will be registration order yes
but don't hold us to it 😉
if you want to be sure, just make a composite listener
then your composite listener can call them in order
j
insert mind-blown emoji here
this is far and away more advanced than junit 😂 I knew this was the right call
s
yeah it's using a proper functional programming approach (where possible, there's vars under the hood, and magic dsl at play too)
so it's way way more powerful than shoving annotations on a few things
And if you need to go to 11, you can use extensions, which give you full control over the engine, so you can do around advice. You can override test results, decide if tests get invoked and so on
j
😈
for now, Tags and EnabledIf are really great for what I need
s
Yes enabled if, tags, listeners, are more than powerful enough for 99% of cases, and then the extensions for when you need to go to 11.
j
so I could implement the MR at the top with the TestCaseExtension? (run my suspend funs in "mostly parallel")
s
the test case extension is good if you want to do something like
Copy code
code here
runTest()
code here
j
gotcha, so I'm still at the mercy of the engine currently running each test case in order and serially until the feature is added
s
they're more for doing plugins and stuff, but you can use them for whatever
until the feature we discussed other day yes
in the meantime just set parallelism to like 10 and use threads for it
it's not coroutines and a big hammer, but would do a job for now
j
right 👍
but first, rip out junit everywhere 😂
s
yep
that's a fun job though, I quite enjoy things like that
j
same here, it's nice seeing how much better you can make things when you use better tools
👍🏻 1
3 Views
Open in Slack
PreviousNext
Powered by