https://kotlinlang.org logo
Title
k

kevin.cianfarini

02/25/2023, 3:40 PM
I'm curious what the proper way to test that a piece of code is delaying for the right amount of time is with the test scheduler. I originally thought that it was to use
advanceTimeBy
but since delays are skipped, using it doesn't make a difference. Is testing that
currentTime
has been incremented as you expect the proper approach?
d

Dmitry Khalanskiy [JB]

02/25/2023, 5:15 PM
You could use the
TimeSource
that tracks the virtual time, with all the usual methods available on it, as in https://github.com/Kotlin/kotlinx.coroutines/blob/master/kotlinx-coroutines-test/common/test/TestCoroutineSchedulerTest.kt#L317-L319
k

kevin.cianfarini

02/25/2023, 5:39 PM
Thanks. I'm curious -- what's the use case for
advanceTimeBy
if delays are always skipped?
d

Dmitry Khalanskiy [JB]

02/25/2023, 5:49 PM
Occasionally, it is quite useful, but yes, most tests never need it. Example where this might be handy: let's say a function makes two network calls. You want to test what happens just after the first network call finishes but before the next one does, without modifying the function under test. Then, you could mock the network calls with
delay(50.milliseconds)
or something and call
advanceTimeBy(51.milliseconds)
in the test body.
k

kevin.cianfarini

02/25/2023, 6:12 PM
Well, if delays are always skipped does that actually do anything?
d

Dmitry Khalanskiy [JB]

02/25/2023, 6:20 PM
Sorry, I don't understand the way to answer this without just repeating the documentation: it runs the scheduled tasks for the given amount of virtual time.
runTest {
  launch {
    delay(50)
    println("2")
    delay(50)
    println("4")
  }
  println("1")
  advanceTimeBy(51)
  println("3")
}
Delay skipping doesn't mean that
delay(n)
is ignored completely; it just means that it doesn't actually perform the delay. The relative order of tasks that use virtual time does obey the delays.
r

Robert Williams

02/27/2023, 10:23 AM
Can someone explain the problem/ provide an example? I always assumed
advanceTimeBy
was designed exactly for this, never had problems and is the recommended approach in the README
k

kevin.cianfarini

02/27/2023, 3:01 PM
Here’s where my understanding is potentially flawed.
suspend fun foo() {
  delay(10_000)
}

@Test fun testFoo() = runTest {
  foo()
  println(currentTime) // Prints 10_000
}
In the above test, the function
testFoo
will automatically skip the delay in
foo
. If you inspect
TestScheduler.currentTime
after the invocation to
foo
it will be
10_000
. Now, how does
advanceTimeBy
play into this? What if I did the following?
@Test fun testFooDifferently() = runTest {
  foo()
  advanceTimeBy(1L)
  assertFooIsStillSuspended() // Pretend this exists 
  advanceTimeBy(9999L)
  assertFooIsFinished() // Pretend this exists, too. 
}
In the above function, since I have the ability to manually control virtual time, I would expect
foo
to still be suspended when the call to
assertFooIsStillSuspended
is called. Because delay skipping is unconditionally applied everywhere the delay immediately gets skipped and
foo
is not in fact still suspended when that assertion is made. It has already completed. This is a scenario I’m currently running into which makes me question the validity of unconditionally applying delay skipping in the test scheduler.
d

Dmitry Khalanskiy [JB]

02/27/2023, 3:03 PM
Function calls are not some magic things in this case, so the code is equivalent to
@Test fun testFooDifferently() = runTest {
  delay(10_000)
  advanceTimeBy(1L)
  assertFooIsStillSuspended() // Can't even exist
  advanceTimeBy(9999L)
  assertFooIsFinished() // Also can't exist
}
suspend
functions are not different from normal ones in the regard that
advanceTimeBy(1L)
can only run after
delay(10_000)
has finished.
k

kevin.cianfarini

02/27/2023, 3:04 PM
So does delay skipping get “paused” if a call to
advanceTimeBy
happens before a delay call is queued in the dispatcher?
The real test I’m working with here is with a flow and I’m asserting whether or not elements get emitted.
I could also amend the above test to
launch
the delay and assert that the job is still active.
d

Dmitry Khalanskiy [JB]

02/27/2023, 3:10 PM
So does delay skipping get “paused”
I don't understand at all what it means to "pause" delay skipping. Make delays take real time? No, this doesn't happen.
if a call to
advanceTimeBy
happens before a delay call
Like this?
runTest {
  advanceTimeBy(1000)
  delay(1000)
}
then, first,
advanceTimeBy
spins pending tasks for one second of virtual time (making the current virtual time 1000), and then,
delay
waits for another second of virtual time (making the current virtual time 2000).
The real test I’m working with here is with a flow and I’m asserting whether or not elements get emitted.
I don't see the relation to
advanceTimeBy
and not sure whether the advice makes sense, but you could try the Turbine library in general for testing flows.
I could also amend the above test to
launch
the delay and assert that the job is still active.
Then it would be completely different.
launch
puts the work into a queue, it does not immediately run its body before returning. I provided an example of
advanceTimeBy
with
launch
above, and there is one in the README as well.
k

kevin.cianfarini

02/27/2023, 3:25 PM
From the documentation this is the part that really confuses me —
fun testFoo() = runTest {
    launch {
        println(1)   // executes during runCurrent()
        delay(1_000) // suspends until time is advanced by at least 1_000
“suspends until time is advanced by at least
1_000
.” Why aren’t these delays skipped and
currentTime
incremented for us? Under what conditions does delay skipping magic happen for us, and under what conditions do we need to do it manually via
advanceTimeBy
?
d

Dmitry Khalanskiy [JB]

02/27/2023, 3:29 PM
Why aren’t these delays skipped and
currentTime
incremented for us?
We want to support interleaving between tasks.
runTest {
  launch {
    delay(50)
    println("2")
    delay(100)
    println("4")
  }
  launch {
    delay(10)
    println("1")
    delay(100)
    println("3")
  }
}
So,
delay
calls must actually suspend the coroutine, instead of being stepped over.
Under what conditions does delay skipping magic happen
There's a queue of tasks, and when the only tasks left are the delayed ones, the delays needed to get to those tasks get skipped.
under what conditions do we need to do it manually via
advanceTimeBy
?
Under very rare ones. I'd say that, when you actually need
advanceTimeBy
, you'll know it. As I said in one of the first messages, it's only occasionally useful.
k

kevin.cianfarini

02/27/2023, 3:32 PM
when the only tasks left are the delayed ones, the delays needed to get to those tasks get skipped.
Is this called out in the documentation anywhere? If so I must have missed it. It’s really unintuitive and I know I’m not the only one operating under the assumption that virtual time is completely manual via
advanceTimeBy
(which is not the case).
d

Dmitry Khalanskiy [JB]

02/27/2023, 3:34 PM
Maybe you used the test framework before the big rework (where it was
runBlockingTest
)? Then, time control was much more prominent: using
TestCoroutineScope
, one had to call
advanceTimeBy
etc manualy.
k

kevin.cianfarini

02/27/2023, 3:36 PM
That is when I last had to use virtual time, yes.
b

Bill Phillips

02/27/2023, 4:34 PM
I’d say that, when you actually need
advanceTimeBy
, you’ll know it.
That’s the question I have: as a tester, I think, “The code I am testing has some time-dependent functionality. That means I need to control the clock”. And every time I have ever had code with time-dependent functionality, that code has been time-dependent due to calls to
delay
So: that is confusing. Because the answer here seems to say, “If your code uses
delay
to implement time-based functionality, then the standard test scheduler’s virtual time and
advanceTimeBy
will not solve your problem”
Related question: when is
advanceTimeBy
needed? What is reliant on virtual time, other than
delay
?
And actually the most important question, then, is “If the code I am testing is uses
delay
for time-dependent functionality, what is the appropriate scheduler setup in my unit test? How do I test, e.g “X did not happen because my timer has not expired yet”?
r

Robert Williams

02/27/2023, 4:56 PM
Good explanation here of “when the delay skipping magic happens” (this is before the runTest refactor but I think it’s mostly the same)
d

Dmitry Khalanskiy [JB]

02/27/2023, 5:01 PM
Because the answer here seems to say, “If your code uses
delay
to implement time-based functionality, then the standard test scheduler’s virtual time and
advanceTimeBy
will not solve your problem”
It does not say that. It says that usually, automatic delay skipping is enough and fine-grained time control is only rarely needed.
when is
advanceTimeBy
needed?
See an example I provided at the top of this thread.
What is reliant on virtual time, other than
delay
?
withTimeout
.
what is the appropriate scheduler setup in my unit test?
What do you mean? We don't provide much configurability, there's barely anything to set up, everything should work out of the box.
How do I test, e.g “X did not happen because my timer has not expired yet”?
Without an idea what your code looks like, tough to say but in general, it could look something like
runTest {
  val time = 10.seconds
  startTimer(time)
  delay(9.seconds)
  assertTimerNotFired()
  delay(1.seconds)
  assertTimerFired()
}
r

Robert Williams

02/27/2023, 5:05 PM
Why would you use
delay
there rather than
advanceTimeBy
? I guess they’re equivalent because of the delay skipping but advanceTimeBy seems more descriptive for a test 🤔
d

Dmitry Khalanskiy [JB]

02/27/2023, 5:10 PM
I prefer
delay
because this way, tests look much simpler. You don't have to know anything about time control: in fact, if you replace
runTest
with
runBlocking
, you get some code that would work the same way, only the delays will take 10 seconds of real time and be non-deterministic. In short, the principle of least power.
Why would one make the reader of the test also learn about
advanceTimeBy
and such when a simple
delay
that works everywhere also works well here?
r

Robert Williams

02/27/2023, 5:15 PM
I get that the point of delay skipping is exactly so that stuff like this just works, I’ve just never seen it recommended before (including in the official documentation I posted above)
Maybe this means we really don’t need advanceTimeBy
d

Dmitry Khalanskiy [JB]

02/27/2023, 5:17 PM
Recommended for what? In the README you linked, the only mention of
advanceTimeBy
is in the "controlling virtual time" section. So, you have to explicitly want this power to search for this.
Also, some people do actually need
advanceTimeBy
. For example, if you injected
TestScope
somewhere and want to spin some coroutines before the test starts, there's really no other choice. Also,
advanceTimeBy
is guaranteed to stop processing tasks once it reaches the target time but before processing any task scheduled for exactly the target time, which is important if you want to test an exact execution order ("first, task A runs for 50 milleseconds, then, task B runs for 2 milleseconds", etc). This is difficult to ensure with `delay`: if several `delay`s are scheduled for the same moment, it's up to the test scheduler to choose which one of them runs first. As I said above, if actually you need it, you know you need it.
k

kevin.cianfarini

02/27/2023, 5:25 PM
As I said above, if actually you need it, you know you need it.
It’s my opinion that more clear examples of how virtual time interacts with (or doesn’t interact with) delay, withTimeOut, timers, etc. It’s unclear when “you know you need it”. I thought I needed it but turns out I didn’t.
Also for what it’s worth, this piece of documentation
If several coroutines are waiting to be executed next, the one scheduled after the smallest delay will be chosen. The virtual time will automatically advance to the point of its resumption.
Doesn’t as clearly explain the virtual time behavior as you explained it earlier.
There’s a queue of tasks, and when the only tasks left are the delayed ones, the delays needed to get to those tasks get skipped.
d

Dmitry Khalanskiy [JB]

02/27/2023, 5:32 PM
more clear examples of how virtual time interacts with (or doesn’t interact with) delay, withTimeOut, timers, etc.
I honestly don't know how to do it better than what we did in the README: there are examples with both
delay
and
withTimeout
that demonstrate all there is to it. If you can suggest new good examples or improvements for the older ones, please do, we accept pull requests for the documentation.
b

Bill Phillips

02/27/2023, 5:40 PM
I’m digging into your example, and… it is very interesting, because if I play around with it in simple test scenarios, I think… “What is Kevin and I’s problem? This works perfectly”
runTest {
        var counter = 0
        backgroundScope.launch {
            counter++
            delay(50.milliseconds)
            counter++
            delay(50.milliseconds)
            counter++
        }
        runCurrent()
        assertEquals(1,counter)
        advanceTimeBy(51.milliseconds.inWholeMilliseconds)
        assertEquals(2, counter)
        advanceTimeBy(50.milliseconds.inWholeMilliseconds)
        assertEquals(3, counter)
    }
this code looks like exactly what we are looking for so that may be the origin of our confusion here: the problems kevin and I are having occur because our tests are not as simple as the above
will try and follow up here with some code that illustrates the sort of test authorship problems we’re having 👍
one thing that is definitely in the mix here is that we use a lot of Turbine: the most common testing pattern for us is to write a fake that does something like this:
class MyFakeService {
  val serviceRequests = Turbine<Request>()
  val serviceResponses = Turbine<Response>()

  suspend fun callService() {
    serviceRequests += Request()
    return serviceResponses.awaitItem()
  }
}
and then write code in our test that looks something like this:
runTest {
  val service = MyFakeService()
  val subject = MySubject(service)
  
  backgroundScope.launch { subject.interactWithService() }

  service.serviceRequests.awaitItem()
  assertThat(subject.someObservableState).isEqualTo(EXPECTED_VALUE_BEFORE_RESPONSE)
  service.serviceResponses += Response()
  
  // do more testing
}
so we are relying on the dispatcher a LOT more than the simple example here