https://kotlinlang.org logo
Title
d

dave08

04/18/2021, 2:26 PM
I wonder if there are any plans in the near future to allow given/and/when/and/then/and tests at the same nesting level... I think this was discussed before as not being easy, but it keeps on eating at me how much nicer the tests could look if it would be possible...
s

sam

04/18/2021, 2:27 PM
Example?
d

dave08

04/18/2021, 2:28 PM
Like:
given {

}.and {

}.when {

}.and {
...
The chaining could give a hint to the framework which level we're holding in and if it's a leaf or a test...
s

sam

04/18/2021, 2:30 PM
And would these nest in output like currently or be consided sibling tests
d

dave08

04/18/2021, 2:31 PM
Well, it would be better the latter (but a bit hacky...), but the former would still avoid the nested testing code which is already a big improvement... it might not always make sense in the output though...
s

sam

04/18/2021, 2:32 PM
The latter is easy to do without the dot. You could just list tests vertically
d

dave08

04/18/2021, 2:32 PM
I wonder how Spock does it? I never used it, but it has it all on one level.
s

sam

04/18/2021, 2:33 PM
Spock is able to use groovys dynamic nature for these tricks
d

dave08

04/18/2021, 2:33 PM
But that would mess up the class setup (to list them vertically...)
s

sam

04/18/2021, 2:33 PM
But then it means using groovy 😉
Would it
Is it much different to what you pasted
d

dave08

04/18/2021, 2:35 PM
Well, what I pasted could potentially save some kind of fixture object that is passed through the chain. Which would allow the setup of the test
Class level objects could always be used, but they're a bit more dangerous, we'd need to assure that they're always reset.
s

sam

04/18/2021, 2:38 PM
ah ok
d

dave08

04/18/2021, 2:38 PM
Minutest did it that way, but they don't have all the features that kotest has... https://github.com/dmcg/minutest#structure-tests
s

sam

04/18/2021, 2:38 PM
I was thinking something like:
val fixture = ...

given("qweqwe") {
}
when("qeqw") {
}
then("werewr") {
}
d

dave08

04/18/2021, 2:39 PM
That puts the fixture outside the given... especially when it's immutable, then the given block might end up being only half the setup
There might even be a provided class in the when context for setting a
result
, and a provided one in the given for setting some fixture objects...
given("qweqwe") { // 'this' is a GivenContext with a var for fixtures
    fixture[1] = RepoStub()
    fixture[0] = SutClass(fixture[1])
}
when("qeqw") { // 'this' is a WhenContext with the fixtures in GivenContext and a result var
    result = fixture[0].methodUnderTest()
}
then("werewr") { // 'this' has all the above
    result shouldBe fixture[1].stubValue
}
maybe something like this?
fixture[0]
and
fixture[1]
might not be so nice, though, but there might be better ways.
s

sam

04/18/2021, 2:49 PM
We could probably pass out through like you want
Can you create a ticket and I'll experiment
d

dave08

04/18/2021, 2:50 PM
Sure, thanks!
s

sam

04/18/2021, 2:56 PM
Great thanks
c

christophsturm

04/18/2021, 5:28 PM
can you show some examples of tests that use this syntax and show why its better?
d

dave08

04/19/2021, 9:59 AM
I have tests with complicated situations and interactions (more like legacy code that I need to test and refactor), and having all the setup description on one
given
line or a bunch of nested `and`s gets pretty messy. This proposition wouldn't be a replacement for the current BehaviourSpec, but rather an optional addition to it... @christophsturm
I guess it's less for code built with SRP in mind...
c

christophsturm

04/19/2021, 10:02 AM
ah thats interesting. I'm thinking a lot about what the perfect testing syntax looks like, but i guess i have only been thinking of tests for well structured code.
s

sam

04/19/2021, 10:03 AM
It gets tricky when a project grows because everyone has their own preferred syntax and it's a balancing act of keeping everyone happy vs kitchen sink of features.
d

dave08

04/19/2021, 10:06 AM
Yup, I was hesitant to propose this 😊... but when I found myself thinking about it in a bunch of occasions, I wondered if others might also need such a thing. Legacy spaghetti code can lead to a complex test suite as it is... (until slowly refactored)
But I wouldn't mind if there was a better idea for this... I personally have a conflict whether everything should be spelled out in the test descriptions or some things could count on the test code to show...
c

christophsturm

04/19/2021, 10:09 AM
one problem is that if you have all those features to support your old legacy code, people will not know what features are ok to use for new code
so they will keep writing too complicated tests even for new features
d

dave08

04/19/2021, 10:10 AM
It's not many features... it's just a certain process that too many components were handling differently, and that currently has too many possible states that have to be handled...
s

sam

04/19/2021, 10:10 AM
I'm not sure Dave's proposal is even possible to support. Because a test in kotest is a function TestContext -> Unit. So you need to pass the test context into the "next" builder, but you can only do that when you're inside the lambda. So chaining these calls might not be possible.
But I will explore it.
c

christophsturm

04/19/2021, 10:12 AM
ah sorry i was not really talking about your proposal but in general. for me the perfect test runner has not many features and just one testing style.
s

sam

04/19/2021, 10:13 AM
Just wait until someone asks for a feature in failfast that you don't need. Do you add it to expand the functionality, or do you say no 🙂
😊 2
c

christophsturm

04/19/2021, 10:14 AM
nobody uses it anyway so thats not going to happen 🙂
s

sam

04/19/2021, 10:14 AM
well you never know
d

dave08

04/19/2021, 10:15 AM
Yeah, I guess kotest already has quite a few things that you don't personally ever use @sam... 😅
s

sam

04/19/2021, 10:15 AM
absolutely, if I had my way I'd delete behavior spec, I think its awful 😂
d

dave08

04/19/2021, 10:15 AM
So what do you use?
s

sam

04/19/2021, 10:15 AM
I only ever use FunSpec
d

dave08

04/19/2021, 10:16 AM
Wow, not even FreeSpec?
s

sam

04/19/2021, 10:16 AM
nope
c

christophsturm

04/19/2021, 10:16 AM
I didnt know that you know about failfast, sam, did you look at it?
s

sam

04/19/2021, 10:16 AM
I did
c

christophsturm

04/19/2021, 10:16 AM
I did not want to advertise it here, and also I want to make sure it works for all my use cases before advertising it more
s

sam

04/19/2021, 10:18 AM
Feel free to advertise it here, I'm not precious about it. If it offers some functionality for people that kotest doesn't, then great, that helps kotlin in general.
c

christophsturm

04/19/2021, 10:19 AM
it sure must look like i have severe nih syndrome. I started writing an orm then i thought the tests run not fast enough then i started writing a test runner. and this weekend i started writing a rest web server because i was frustrated with ktor, and http4k does not support coroutines
s

sam

04/19/2021, 10:20 AM
http4k not supporting coroutines is incredibly frustrating. I would use it tomorrow if it had coroutine support, but they seem to think it doesn't matter (if you follow the threads, no pun intended). I think that's mistake personally.
c

christophsturm

04/19/2021, 10:20 AM
hehe
s

sam

04/19/2021, 10:21 AM
It seems easy to make a test framework, and when it's focused on your own use cases it probably is easy. There's some "bugs" that people bring up time after time and they're out of kotest's control. The classic one about parent tests being included in test counts - that's just how gradle works. And gradle has bugs in its testing implementation that have remained open for years.
c

christophsturm

04/19/2021, 10:22 AM
in my web server handlers look like this:
class UserService : RestService {
    suspend fun create(user: User): User {
        delay(1)
        return user.copy(id = "userId")
    }

    suspend fun show(userId: Int): User {
        delay(1)
        return User(id = userId.toString(), name = "User $userId")
    }

}
that fits 99% of the code that i write and makes it really easy to test
in failfast i now support junit engine for idea support, but in gradle i still only run the tests as a main method.
s

sam

04/19/2021, 10:23 AM
The problem with NIH is that it's not so bad doing the simple stuff, but when you need some rock solid handling of url encoding, or websockets, or whatever, you're really going to spend a lot of time on what has been done before.
c

christophsturm

04/19/2021, 10:23 AM
hmm yeah I did not really write it from scratch, i use undertow. like i did in my first kotlin web projects
s

sam

04/19/2021, 10:24 AM
undertow is good, I built some scala stuff on it ages ago
c

christophsturm

04/19/2021, 10:24 AM
it seems that it does not have much momentum, all the redhat server side stuff seems to focus on quarkus now
but it still gets security updates and has no bugs that bother me
so i thought i start with something i know
s

sam

04/19/2021, 10:26 AM
what didn't you like about ktor?
c

christophsturm

04/19/2021, 10:27 AM
error handling. sometimes things just dont work and the error message is not useful, and trying to debug it is very complicated
s

sam

04/19/2021, 10:28 AM
yeah for sure the phase / install parts of it are odd
c

christophsturm

04/19/2021, 10:28 AM
it does not feel robust at all. only the happy path is really tested
s

sam

04/19/2021, 10:29 AM
I know we had an issue upgrading from 1.4.2 to 1.5.0 which broke the CORS support
had to roll back
c

christophsturm

04/19/2021, 10:30 AM
when you sent an authorization header to it that is misspelled you get an internal server error with a huge stacktrace that says nothing about the problem. or if you send json that does not have the expected format. thats both use cases that happen really often.
I want small libs with good testing coverage
s

sam

04/19/2021, 10:31 AM
tested with failfast of course 🙂
c

christophsturm

04/19/2021, 10:32 AM
or kotest!
😂 1
so what did you think of failfast?
s

sam

04/19/2021, 10:35 AM
very nicely implemented
I liked the configuring which tests to run programatically. That sits nicely with my functional programming mindset.
I didn't delve deeply enough to look at the auto test stuff, but I like that idea. SBT in scalaland does a good job of that and I quite liked it there.
c

christophsturm

04/19/2021, 10:37 AM
oh nice thank you
the autotest stuff does not yet work as well as it should. but i did find out that you can find dependent classes by reading just the beginning of the class file (constant pool)
i thought that it would be enought to just run updated class files, because kotlin would recompile tests that use a class when the class changes. but kotlin is too clever for that and recompiles only when the interface changes
s

sam

04/19/2021, 10:39 AM
look at classgraph
c

christophsturm

04/19/2021, 10:39 AM
yeah classgraph can do it. and seems to be really fast.
s

sam

04/19/2021, 10:40 AM
yeah it can scan quickly
c

christophsturm

04/19/2021, 10:40 AM
i just try to keep the code that runs single threaded really small and with no dependencies
as soon as i have found a single test to run its much more relaxed, but i want all cores to work from the beginning
s

sam

04/19/2021, 10:40 AM
when you get it working flawlessly, do a PR to kotest 🙂
c

christophsturm

04/19/2021, 10:42 AM
where would autotest fit best into the kotest architecture?
s

sam

04/19/2021, 10:42 AM
would have to make a gradle task that spins up and keeps running I guess
that's how SBT does it
c

christophsturm

04/19/2021, 10:43 AM
ah right, so the idea plugin and the gradle plugin could support it
s

sam

04/19/2021, 10:43 AM
yeah
kotest gradle plugin that doesn't need junit at all
c

christophsturm

04/19/2021, 10:43 AM
i think if it just works in idea and works there perfectly that would be enough
s

sam

04/19/2021, 10:44 AM
yeah probably
then maybe it could be added to the intellij plugin for kotest
c

christophsturm

04/19/2021, 10:44 AM
I started working on an idea plugin for failfast but there seems to be zero documentation
the best documentation for creating a test runner plugin seems to be the kotest plugin
s

sam

04/19/2021, 10:47 AM
lol
probably is yes
I spent many a night pouring over that
You are putting a lot of effort into recreating the entire kotest ecosystem 🙂
c

christophsturm

04/19/2021, 10:58 AM
it sure looks that way. I just wanted to iterate fast and build exactly the syntax that i wanted. so starting from scratch seemed to be the best solution
s

sam

04/19/2021, 10:59 AM
You will have learned a lot about how junit platform, gradle, etc works while doing that, so feel free to hop in and contribute to kotest in any way you want.
c

christophsturm

04/19/2021, 10:59 AM
i did find a kotest bug that i did not report yet. it seems the pitest plugin has a dependency on pitest, and that leads to using an older pitest version, and you cannot override it in gradle.
s

sam

04/19/2021, 11:01 AM
seems more like a gradle bug, gradle should pick whatever version is latest
I have seen this behavior in gradle quite a bit though - the
implementation
thing doesn't seem super robust
It might be related to being a multi platform project
c

christophsturm

04/19/2021, 11:01 AM
i think the dependency should just be compileOnly
I need to move pitest to its own repo soon so I will do it then, before 4.5 release
c

christophsturm

04/19/2021, 11:04 AM
lol sorry i should have just created a ticket.
s

sam

04/19/2021, 11:04 AM
that's fine, took me 2 seconds
I wonder if you took the KotestEngineLauncher, programatically registered tests like in failfast, if it would run the tests fast enough for you
KotestEngineLauncher()
   .withSpec(DummySpec1::class)
   .withSpec(DummySpec2::class)
   .launch()
c

christophsturm

04/19/2021, 11:15 AM
i was thinking of a queue based design where the discovery starts putting tests into the queue, and execution starts as soon as there is one test in the queue
s

sam

04/19/2021, 11:16 AM
might save you a second, depends how slow discovery is.
c

christophsturm

04/19/2021, 11:18 AM
the motto of failfast is "with enough cores any test suite can run in one second"
s

sam

04/19/2021, 11:18 AM
lol well Kotest has 4000+ tests
good luck running that in < 1 second
c

christophsturm

04/19/2021, 11:19 AM
yeah its more theoretical. and it only works if you have a lot of short tests.
my orm test suite breaks that expectation and takes 10 seconds
s

sam

04/19/2021, 11:20 AM
right
there's some tests in kotest that are testing Thread.sleeps so they'll never run fast
c

christophsturm

04/19/2021, 11:20 AM
why do they do that?
coroutines are really great for multi threaded test running, if a test needs a dependency that takes some time to spin up i do it in a separate io scheduler to make the cpu core available to a different test.
s

sam

04/19/2021, 11:22 AM
to test concurrency works for both suspension and blocking
saying "you should be using coroutines" is not adequate for people who can't / don't use suspension
c

christophsturm

04/19/2021, 11:22 AM
sure
s

sam

04/19/2021, 11:23 AM
so for things like the timeout functionality, you want to test that a timeout works for both blocking and suspension
"Test".config(timeout = 300.milliseconds) {
  Thread.sleep(100)
}
c

christophsturm

04/19/2021, 11:23 AM
you will always have a smoke test suite that is slower.
s

sam

04/19/2021, 11:23 AM
"Test 2".config(timeout = 300.milliseconds) {
  Thread.sleep(500) // should fail
}
c

christophsturm

04/19/2021, 11:24 AM
hmm tests like that only work single threaded, or the first test will also fail if you have too much load
s

sam

04/19/2021, 11:25 AM
yes, so you need to put in a bit of "wiggle" room
that's why I don't do:
"Test".config(timeout = 5.milliseconds) {
  Thread.sleep(4)
}