Differentiating between latency-critical operation...

# coroutines

[CLOVIS]

Differentiating between latency-critical operations and non-latency-critical operations

In large systems, we often have two different categories of operations:
• Latency-critical: answering a request from the user…
• Non-latency-critical: async events…
We have code like:

suspend fun doSomething() = coroutineScope {
    val data1 = async { fetchData1() }
    val data2 = async { fetchData2() }
    val data3 = async { fetchData3() }

    doSomething(data1.await(), data2.await(), data3.await())
}

Is there a way to declare that this function should use high parallelism when it's latency-critical, and use parallelism only if the system is idle? We don't want to simply reduce the parallelism of non-critical operations, because if the system is doing few things, having high parallelism for these operations is still beneficial.

Maybe some kind of priority? Maybe with another dispatcher?

[Sam]

Which resource is under contention? CPU, or network + remote service?

[Dmitry Khalanskiy [JB]]

If I understood this correctly, I think a dispatcher would solve this issue. Something similar to https://github.com/Kotlin/kotlinx.coroutines/blob/master/kotlinx-coroutines-core/common/src/internal/LimitedDispatcher.kt, but allocating new workers not just on demand but when some condition is satisfied (for example, either the queue has too many elements or the coroutineContext has a special "urgent operation" marker).

[CLOVIS]

@Sam CPU.

[louiscad]

Dispatchers.Default.limitedParallelism and/or differentiated use of yield() in between CPU intensive loops could help you do that

[CLOVIS]

@louiscad I don't want to set a limit to parallelism though. I'd like something like Dispatchers.Default.lowPriority

[louiscad]

The JVM has a concept of thread priority

[louiscad]

I wish Roman Elizarov was still there to provide some guidance on using this right.

[CLOVIS]

I'm not sure this use-case warrants creating new threads 🤔

[Uli Bubenheimer]

Here is a related Github issue from a couple years ago, it has other related issues linked: https://github.com/Kotlin/kotlinx.coroutines/issues/3178

[louiscad]

Pretty sure you could make a dispatcher wrapper that sets and resets the priority for each dispatch it receives.

[louiscad]

try+finally is your friend

[CLOVIS]

You mean the OS priority? I think that is quite expensive to set

[louiscad]

Did you read something about that being expensive?

On the JVM, you can do: Thread.currentThread().priority = something

[CLOVIS]

I'm not talking about code size, I'm talking about execution speed. Thread priority is an input into the OS scheduler, and changing its settings on each coroutine dispatch will probably be more expensive than what the coroutines do themselves…

[louiscad]

I wasn't talking about code size either 🙂

You're saying "probably more expensive", so it's all assumptions… I'd test it. You can change the priority many times, and measure the time it took in nanos

[Jarkko Miettinen]

Setting thread priority is relatively expensive if it alters thread's priority as you'll end up doing a syscall. Syscall costs thousands of CPU cycles and affects your L1/L2 caches and TLB.

(Above assumes Linux)

[Jarkko Miettinen]

Neither coroutines nor JVM virtual threads have pre-emptive scheduling so you'll need to manually check for higher-priority tasks and yield if needed.
Which can be hard if you don't really control your system and instead have a lot of code from external libs on the leaf levels of execution.

Here you'll need to be concerned on priority inversion where a high-prio task actually awaits for low-prio tasks.

[Jarkko Miettinen]

Above also assumes that changing thread priority actually does something.
Whole JVM needs to be scheduled under real-time policy and you must have relevant capacity to alter priorities.