https://kotlinlang.org logo
Join Slack
Powered by
I'm curious what the proper way to test that a pie...
# coroutines
k
I'm curious what the proper way to test that a piece of code is delaying for the right amount of time is with the test scheduler. I originally thought that it was to use 
advanceTimeBy
but since delays are skipped, using it doesn't make a difference. Is testing that 
currentTime
has been incremented as you expect the proper approach?
d
You could use the 
TimeSource
that tracks the virtual time, with all the usual methods available on it, as in https://github.com/Kotlin/kotlinx.coroutines/blob/master/kotlinx-coroutines-test/common/test/TestCoroutineSchedulerTest.kt#L317-L319
k
Thanks. I'm curious -- what's the use case for 
advanceTimeBy
if delays are always skipped?
d
Occasionally, it is quite useful, but yes, most tests never need it. Example where this might be handy: let's say a function makes two network calls. You want to test what happens just after the first network call finishes but before the next one does, without modifying the function under test. Then, you could mock the network calls with 
delay(50.milliseconds)
or something and call 
advanceTimeBy(51.milliseconds)
in the test body.
k
Well, if delays are always skipped does that actually do anything?
d
Sorry, I don't understand the way to answer this without just repeating the documentation: it runs the scheduled tasks for the given amount of virtual time.
Copy code
runTest {
  launch {
    delay(50)
    println("2")
    delay(50)
    println("4")
  }
  println("1")
  advanceTimeBy(51)
  println("3")
}
Delay skipping doesn't mean that 
delay(n)
is ignored completely; it just means that it doesn't actually perform the delay. The relative order of tasks that use virtual time does obey the delays.
r
Can someone explain the problem/ provide an example? I always assumed 
advanceTimeBy
was designed exactly for this, never had problems and is the recommended approach in the README
k
Here’s where my understanding is potentially flawed.
Copy code
suspend fun foo() {
  delay(10_000)
}

@Test fun testFoo() = runTest {
  foo()
  println(currentTime) // Prints 10_000
}
In the above test, the function 
testFoo
will automatically skip the delay in 
foo
. If you inspect 
TestScheduler.currentTime
after the invocation to 
foo
it will be 
10_000
. Now, how does 
advanceTimeBy
play into this? What if I did the following?
Copy code
@Test fun testFooDifferently() = runTest {
  foo()
  advanceTimeBy(1L)
  assertFooIsStillSuspended() // Pretend this exists 
  advanceTimeBy(9999L)
  assertFooIsFinished() // Pretend this exists, too. 
}
In the above function, since I have the ability to manually control virtual time, I would expect 
foo
to still be suspended when the call to 
assertFooIsStillSuspended
is called. Because delay skipping is unconditionally applied everywhere the delay immediately gets skipped and 
foo
is not in fact still suspended when that assertion is made. It has already completed. This is a scenario I’m currently running into which makes me question the validity of unconditionally applying delay skipping in the test scheduler.
d
Function calls are not some magic things in this case, so the code is equivalent to
Copy code
@Test fun testFooDifferently() = runTest {
  delay(10_000)
  advanceTimeBy(1L)
  assertFooIsStillSuspended() // Can't even exist
  advanceTimeBy(9999L)
  assertFooIsFinished() // Also can't exist
}
suspend
functions are not different from normal ones in the regard that 
advanceTimeBy(1L)
can only run after 
delay(10_000)
has finished.
k
So does delay skipping get “paused” if a call to 
advanceTimeBy
happens before a delay call is queued in the dispatcher?
The real test I’m working with here is with a flow and I’m asserting whether or not elements get emitted.
I could also amend the above test to 
launch
the delay and assert that the job is still active.
d
So does delay skipping get “paused”
I don't understand at all what it means to "pause" delay skipping. Make delays take real time? No, this doesn't happen.
if a call to 
advanceTimeBy
happens before a delay call
Like this?
Copy code
runTest {
  advanceTimeBy(1000)
  delay(1000)
}
then, first, 
advanceTimeBy
spins pending tasks for one second of virtual time (making the current virtual time 1000), and then, 
delay
waits for another second of virtual time (making the current virtual time 2000).
The real test I’m working with here is with a flow and I’m asserting whether or not elements get emitted.
I don't see the relation to 
advanceTimeBy
and not sure whether the advice makes sense, but you could try the Turbine library in general for testing flows.
I could also amend the above test to 
launch
the delay and assert that the job is still active.
Then it would be completely different. 
launch
puts the work into a queue, it does not immediately run its body before returning. I provided an example of 
advanceTimeBy
with 
launch
above, and there is one in the README as well.
k
From the documentation this is the part that really confuses me —
Copy code
fun testFoo() = runTest {
    launch {
        println(1)   // executes during runCurrent()
        delay(1_000) // suspends until time is advanced by at least 1_000
“suspends until time is advanced by at least 
1_000
.” Why aren’t these delays skipped and 
currentTime
incremented for us? Under what conditions does delay skipping magic happen for us, and under what conditions do we need to do it manually via 
advanceTimeBy
?
d
Why aren’t these delays skipped and 
currentTime
incremented for us?
We want to support interleaving between tasks.
Copy code
runTest {
  launch {
    delay(50)
    println("2")
    delay(100)
    println("4")
  }
  launch {
    delay(10)
    println("1")
    delay(100)
    println("3")
  }
}
So, 
delay
calls must actually suspend the coroutine, instead of being stepped over.
Under what conditions does delay skipping magic happen
There's a queue of tasks, and when the only tasks left are the delayed ones, the delays needed to get to those tasks get skipped.
under what conditions do we need to do it manually via 
advanceTimeBy
?
Under very rare ones. I'd say that, when you actually need 
advanceTimeBy
, you'll know it. As I said in one of the first messages, it's only occasionally useful.
k
when the only tasks left are the delayed ones, the delays needed to get to those tasks get skipped.
Is this called out in the documentation anywhere? If so I must have missed it. It’s really unintuitive and I know I’m not the only one operating under the assumption that virtual time is completely manual via 
advanceTimeBy
(which is not the case).
➕ 1
d
Maybe you used the test framework before the big rework (where it was 
runBlockingTest
)? Then, time control was much more prominent: using 
TestCoroutineScope
, one had to call 
advanceTimeBy
etc manualy.
k
That is when I last had to use virtual time, yes.
b
I’d say that, when you actually need 
advanceTimeBy
, you’ll know it.
That’s the question I have: as a tester, I think, “The code I am testing has some time-dependent functionality. That means I need to control the clock”. And every time I have ever had code with time-dependent functionality, that code has been time-dependent due to calls to 
delay
So: that is confusing. Because the answer here seems to say, “If your code uses 
delay
to implement time-based functionality, then the standard test scheduler’s virtual time and 
advanceTimeBy
will not solve your problem”
Related question: when is 
advanceTimeBy
needed? What is reliant on virtual time, other than 
delay
?
And actually the most important question, then, is “If the code I am testing is uses 
delay
for time-dependent functionality, what is the appropriate scheduler setup in my unit test? How do I test, e.g “X did not happen because my timer has not expired yet”?
r
Good explanation here of “when the delay skipping magic happens” (this is before the runTest refactor but I think it’s mostly the same)
d
Because the answer here seems to say, “If your code uses 
delay
to implement time-based functionality, then the standard test scheduler’s virtual time and 
advanceTimeBy
will not solve your problem”
It does not say that. It says that usually, automatic delay skipping is enough and fine-grained time control is only rarely needed.
when is 
advanceTimeBy
needed?
See an example I provided at the top of this thread.
What is reliant on virtual time, other than 
delay
?
withTimeout
.
what is the appropriate scheduler setup in my unit test?
What do you mean? We don't provide much configurability, there's barely anything to set up, everything should work out of the box.
How do I test, e.g “X did not happen because my timer has not expired yet”?
Without an idea what your code looks like, tough to say but in general, it could look something like
Copy code
runTest {
  val time = 10.seconds
  startTimer(time)
  delay(9.seconds)
  assertTimerNotFired()
  delay(1.seconds)
  assertTimerFired()
}
r
Why would you use 
delay
there rather than 
advanceTimeBy
? I guess they’re equivalent because of the delay skipping but advanceTimeBy seems more descriptive for a test 🤔
d
I prefer 
delay
because this way, tests look much simpler. You don't have to know anything about time control: in fact, if you replace 
runTest
with 
runBlocking
, you get some code that would work the same way, only the delays will take 10 seconds of real time and be non-deterministic. In short, the principle of least power.
Why would one make the reader of the test also learn about 
advanceTimeBy
and such when a simple 
delay
that works everywhere also works well here?
r
I get that the point of delay skipping is exactly so that stuff like this just works, I’ve just never seen it recommended before (including in the official documentation I posted above)
Maybe this means we really don’t need advanceTimeBy
d
Recommended for what? In the README you linked, the only mention of 
advanceTimeBy
is in the "controlling virtual time" section. So, you have to explicitly want this power to search for this.
Also, some people do actually need 
advanceTimeBy
. For example, if you injected 
TestScope
somewhere and want to spin some coroutines before the test starts, there's really no other choice. Also, 
advanceTimeBy
is guaranteed to stop processing tasks once it reaches the target time but before processing any task scheduled for exactly the target time, which is important if you want to test an exact execution order ("first, task A runs for 50 milleseconds, then, task B runs for 2 milleseconds", etc). This is difficult to ensure with `delay`: if several `delay`s are scheduled for the same moment, it's up to the test scheduler to choose which one of them runs first. As I said above, if actually you need it, you know you need it.
k
As I said above, if actually you need it, you know you need it.
It’s my opinion that more clear examples of how virtual time interacts with (or doesn’t interact with) delay, withTimeOut, timers, etc. It’s unclear when “you know you need it”. I thought I needed it but turns out I didn’t.
➕ 1
Also for what it’s worth, this piece of documentation
If several coroutines are waiting to be executed next, the one scheduled after the smallest delay will be chosen. The virtual time will automatically advance to the point of its resumption.
Doesn’t as clearly explain the virtual time behavior as you explained it earlier.
There’s a queue of tasks, and when the only tasks left are the delayed ones, the delays needed to get to those tasks get skipped.
d
more clear examples of how virtual time interacts with (or doesn’t interact with) delay, withTimeOut, timers, etc.
I honestly don't know how to do it better than what we did in the README: there are examples with both 
delay
and 
withTimeout
that demonstrate all there is to it. If you can suggest new good examples or improvements for the older ones, please do, we accept pull requests for the documentation.
b
I’m digging into your example, and… it is very interesting, because if I play around with it in simple test scenarios, I think… “What is Kevin and I’s problem? This works perfectly”
Copy code
runTest {
        var counter = 0
        backgroundScope.launch {
            counter++
            delay(50.milliseconds)
            counter++
            delay(50.milliseconds)
            counter++
        }
        runCurrent()
        assertEquals(1,counter)
        advanceTimeBy(51.milliseconds.inWholeMilliseconds)
        assertEquals(2, counter)
        advanceTimeBy(50.milliseconds.inWholeMilliseconds)
        assertEquals(3, counter)
    }
this code looks like exactly what we are looking for so that may be the origin of our confusion here: the problems kevin and I are having occur because our tests are not as simple as the above
will try and follow up here with some code that illustrates the sort of test authorship problems we’re having 👍
one thing that is definitely in the mix here is that we use a lot of Turbine: the most common testing pattern for us is to write a fake that does something like this:
Copy code
class MyFakeService {
  val serviceRequests = Turbine<Request>()
  val serviceResponses = Turbine<Response>()

  suspend fun callService() {
    serviceRequests += Request()
    return serviceResponses.awaitItem()
  }
}
and then write code in our test that looks something like this:
Copy code
runTest {
  val service = MyFakeService()
  val subject = MySubject(service)
  
  backgroundScope.launch { subject.interactWithService() }

  service.serviceRequests.awaitItem()
  assertThat(subject.someObservableState).isEqualTo(EXPECTED_VALUE_BEFORE_RESPONSE)
  service.serviceResponses += Response()
  
  // do more testing
}
so we are relying on the dispatcher a LOT more than the simple example here
108 Views
Open in Slack
PreviousNext
Powered by