https://kotlinlang.org logo
Join Slack
Powered by
Hey everyone, :wave: the Kotest here team asking f...
# kotest
o
Hey everyone, 👋 the Kotest here team asking for feedback! Are you using 
eventually
, 
continually
, 
until
or 
retry
in coroutines using the `TestDispatcher`'s virtual time? (You are using virtual time in tests configured with 
coroutineTestScope = true
.) So far, all of those functions were operating on real time, even when used in coroutines operating on virtual time. We intend to change this, so that with the 
TestDispatcher
in use, these functions also use virtual time. So, for example, where a 
delay
would use virtual time, 
eventually
would as well. Switching from virtual to real-time schedulers via 
withContext(<http://Dispatchers.IO|Dispatchers.IO>)
is supported, and would switch to real time for everything. Do you have a use case or can you think of one, where you would mix real time and virtual time for the above scenarios? 👌 I have a use case or can imagine one, details in 🧵 no red I don't have a use case and I have no idea why I'd want to do this. Clarification: Please answer 👌 if and only if both of the following conditions are true 1. you have a coroutine running on virtual time, and 2. in that coroutine you use 
eventually
with a real-time timeout. If you have or can imagine such a concrete use case, please post it in the 🧵.
no red 5
👌 1
k
I don't currently use 
TestDispatcher
but may do in the future. I currently use 
eventually
and rely on real time because it's waiting for a service running in another container to start.
o
So then you would not use 
eventually
on a 
TestDispatcher
, right? Where would the 
TestDispatcher
come in? What does the use case look like?
k
I have never used 
TestDispatcher
so far, so I may be mistaken thinking it might be relevant here. My use case is that my production code contains some coroutines, most of which use 
withContext(<http://Dispatchers.IO|Dispatchers.IO>)
, and I have an integration test that depends on some containers having been started which communicate with the system under test, hence I use 
eventually
as those containers can take some time to start up.
o
Using real-time 
eventually
to wait for something to become ready is perfectly legitimate. You'd just use the 
eventually
within a standard dispatcher to do so. A 
TestDispatcher
requires all of your code under test to execute in a single thread. Why? Because the purpose of the 
TestDispatcher
is to skip time, omitting unnecessary waits, making tests faster. And it can only skip time if it knows precisely, what comes next. So if coroutine #1 delays for 2 seconds, and coroutine #2 one for 1 second, it would just advance virtual time by 1 second (eliminating a real-time wait), then resume coroutine #2. It can never know what comes next in code running on different threads (or just concurrently under non-test dispatchers). Even within a 
TestDispatcher
-controlled coroutine you could still wait for some service to start up by doing 
withContext(<http://Dispatchers.IO|Dispatchers.IO>) { eventually(...) { ... } }
. So there would be no mix of virtual time and real time.
thank you color 1
s
Test Dispatcher I think is a misleading choice of name. It doesn't indicate that it gives you a virtual clock. You have to just know.
o
Yep. That name is too generic.
c
In my case, I always use the virtual time everywhere. I often have to test algorithms that are time-dependent (e.g. cache expiration) using injected clocks. However, I very rarely use 
eventually
& co: because I'm using coroutines, most of the time I can just await the event I want. Or, in the case of cache expiration, I know after how long it's supposed to trigger, so I can simply 
delay
until that point.
o
Maybe the question about mixing virtual and real time was not clear enough (I'll edit the question above correspondingly): Please answer 👌 if and only if both of the following conditions are true 1. you have a coroutine running on virtual time, and 2. in that coroutine you use 
eventually
with a real-time timeout. If you have or can imagine such a concrete use case, please post it. If this additional explanation would change your answer, you can choose a different one at any time.
👍 1
a
I think you and I have independently come to the same conclusion @Oliver.O :) I was thinking about your PR, and I was questioning why utilities to help with flakey tests should be compatible with the virtual time, which cannot be flaky. I really couldn't think of a reason.
o
Yep. I also tried really hard to come up with counter arguments, but so far I couldn't find any. So by asking here I just wanted to make sure that I was not missing something, because people are creative. ☺️
a
Well, I guess I can think of one reason; to allow Kotest to be able to test `eventually`/`continually`/etc in a performant & stable way.
but that's not something that should be user visible
o
That's what we'd do now. But this does not qualify as mixing: Everything runs on virtual time, which means deterministic only. It doesn't hurt if users were allowed to do the same (all on virtual time). Better than accidental mixing in any case.
💯 2
s
This might be opening a can of worms but why would you need to use eventually with virtual time ? eventually is designed for non-deterministic scenarios. Eg I'm writing to a kafka container, and I want to wait until the consumer picks it up. That wouldn't care about a coroutine dispatcher.
o
Sure. We need it for stable internal testing of 
eventually
and others. Kotest users would probably not need it.
a
I think the question could be rephrased: If someone enables virtual time for a test that uses 
eventually
, what would be more surprising? 
eventually
also using virtual time, or not?
👍 1
(maybe 'rephrased' is the wrong word, I didn't mean to imply that your original question needed rephrasing @Oliver.O :))
o
I like that rephrasing. I think it's a very legitimate variant to improve clarity!
s
eventually is for when you can't control time, and some non-deterministic process needs brute force approach. I wouldn't even expose virtual time outside of kotest via it ? If that's possible ?
o
Where would eventually expose virtual time?
s
I guess what I'm asking is, virtual time seems unrelated to what eventually is trying to solve
o
That's true.
s
Virtual time or not, it's not going to change how quickly fsync takes
Or how quickly kafka flushes a consumer
o
Yep. But who are we to tell a user who insists on using virtual time?
s
The concept just makes no sense. I'm using eventually to wait until I receive a message over a socket. If I use virtual time to advance time, that's not going to make any difference to my socket and now eventually breaks
If I'm explaining it properly
o
I actually came up with a case, where a Kotest user could benefit: If they do test infra tests like "I want to use eventually in that configuration and make sure that it behaves in a specific way. I also want to make sure that I have understood how it works and ensure that via a test that uses virtual time for stability". Then it's like me testing BlockHound infra by turning it on and then testing "is this thing on?"
Other than that, what would we prefer? Put in extra code to block virtual time use for external users?
... in conjunction with eventually and others, of course.
(AFK for a while. Get back later.)
a
I agree Sam, it doesn't make sense to me either. Virtual time is used to mock runtime behaviour, while 
eventually
et al is configuring test behaviour.
o
To me it seems like: 
delay
uses time, 
eventually
uses time. Both always use the same time source, whatever it is. Result for now is: We have stable tests. Kotest users might not need it (if we ignore the above case), it doesn't hurt either. If we don't want that, what should we change? What would be the added value?
s
Does this have any effect on users?
o
Using eventually in a coroutine with a regular dispatcher: no change. Using eventually in a coroutine with a test dispatcher: skips time where it would have used real time before.
s
Seems wrong to me but open to whatevrr
o
So far it looks like such use with a test dispatcher would have been a bug anyway. I’d everything before, after and inside eventually is on virtual time, what would be the purpose of real-time eventually?
s
Like I said eventually is used to brute force non deterministic stuff
Mostly network stuff
o
Yes, and as I see it, there’s no change here. As usual, we invoke such stuff on the IO dispatcher. This would still work as before.
a
Let's say someone uses Kotest to test their HttpClient. During tests they spin up a mock server, but for whatever reason, the server is flaky on CI. For that reason, they use 
eventually
with a limit of 1 minute, because it's easier than fixing the flaky server. Their HttpClient implementation has a custom retry, using coroutines and 
delay(...)
. In their HttpClientTest they don't actually want to 
delay()
when the server fails, so they enable virtual-time. However, if 
eventually
respects virtual-time, and the server is flaky, now the HttpClient's 
delay()
will cause the 
eventually
to complete earlier than what they want. What they wanted is to make sure that the tests complete within a fixed time limit. But instead an implementation detail of their HttpClient will mean that the flaky server doesn't get retried as often as they want. What could they do instead?
o
Their HttpClient implementation has a custom retry, using coroutines and 
delay(...)
. In their HttpClientTest they don't actually want to 
delay()
when the server fails, so they enable virtual-time.
What is the delay inside the HTTP client for?
a
1. their HttpClient calls the server 2. server returns an error code 3. HttpClient delays for X amount of time 4. HttpClient retries Y times
o
You couldn't use virtual time and skip delays with that at all. It would just not work: The server responds in real time. If you then don't delay in real time, the server would never be given the (real) time it takes to get back on its feet. Virtual time never works with outside services. It only works inside its own little world on a single thread where you control everything.
a
What about if in production the server needs the backoff, but in tests the backoff isn't needed and it should use virtual time?
o
And the server is still external to the test?
a
what does external to the test mean?
o
Not called in-process. How does it know it is being tested?
I guess in the end it boils down to having an explicit test mode: Parameterize the server, so that it acts differently under test, parameterize the client so that it does not delay. My understanding is: Virtual time is not meant to just short-cut delays. It is meant to retain the sequencing of different coroutines, which is implied by such delays. To do so, you need to have everything depending on that sequencing on virtual time. You can insert real-time stuff anywhere you like, e.g. via 
withContext(<http://Dispatchers.IO|Dispatchers.IO>) { ... }
, but switching back to outside virtual time inside the new real-time world is a no-no.
a
I guess in the end it boils down to having an explicit test mode: Parameterize the server, so that it acts differently under test, parameterize the client so that it does not delay.
What if the code can't be parameterised? E.g. it's in a 3rd party library
o
Injecting a test dispatcher into code beyond your control, without a proper contract? The 3rd party code could introduce a dispatcher, other delays and more changes in the next release. Doesn’t that sound like a perfect footgun? But if you really wanted to, you probably could do something like 
withContext(StandardTestDispatcher()) {
… 
}
inside an 
eventually
running on a regular dispatcher.
28 Views
Open in Slack
PreviousNext
Powered by