https://kotlinlang.org logo
j

Jebus_Chris

12/31/2022, 3:36 PM
What’s the best or proper way to use coroutines with api and database calls? The api is using ktor client (it can suspend) and the db is using jdbc (blocking) with sqldelight+hikari. Say I need 1000 api calls and then 1000 db inserts, right now I just make 1000 coroutines for the api and then 1000 for the db as the api calls return (all using Dispatchers IO). Right now main handles all the coroutines. Should they instead be in the api/db classes themselves or even in both spots? This is just a script, but would the approach change for a web app in ktor? Simplified code below:
Copy code
fun main() {
    runBlocking { 
        launch {
            values.forEach {
                CoroutineScope(coroutineContext + <http://Dispatchers.IO|Dispatchers.IO>).launch {
                    val data = api.getData(it)
                    channel.send(data)
                }
            }
            for (i in 1..values.size) {
                val data = channel.receive()
                CoroutineScope(coroutineContext + <http://Dispatchers.IO|Dispatchers.IO>).launch {
                    db.insert(data)
                }
            }
        }.join()
    }
}
j

jw

12/31/2022, 3:39 PM
I would use a single coroutine for the database and a channel to send the items to insert
j

Jebus_Chris

12/31/2022, 4:02 PM
Wouldn’t that make the database calls execute sequentially?
j

jw

12/31/2022, 4:25 PM
Yes
Write locks are exclusive so even with 1000 coroutines and 1000 threads executing perfectly in parallel it will be sequential
So you might as well have only a single coroutine and eliminate lock contention
j

Jebus_Chris

12/31/2022, 10:48 PM
I see, so if they were reads then it would make sense.
Might be more of a design questions, but should the coroutines be in the main class (or calling class/function) or should they instead be in the api or the db class?
c

Colton Idle

01/01/2023, 3:45 PM
Write locks are exclusive so even with 1000 coroutines and 1000 threads executing perfectly in parallel it will be sequential
wait. maybe dumb question. but is that basically true of all SQL* databases? I never knew that it'd be sequential without anyway to write it. How do backend DBs keep up with tons of writes? 🤯
j

jw

01/01/2023, 5:02 PM
Depends on the DB, but generally you write to a log file which is append only
Then that's applied asynchronously
c

Colton Idle

01/01/2023, 5:53 PM
I should probably read up on how databases work at a fundamental level, but i was reading a topic similar to this which is funny that this came up in kotlinlang slack. as always thanks for teaching.
j

jw

01/01/2023, 6:05 PM
They're a great source of learning because performance and correctness are both critical
l

louiscad

01/02/2023, 12:06 PM
You can replace
CoroutineScope(coroutineContext + <http://Dispatchers.IO|Dispatchers.IO>).launch
by just
launch(<http://Dispatchers.IO|Dispatchers.IO>)
FYI.
j

Jebus_Chris

01/02/2023, 3:11 PM
I remember messing around with that. If I don’t have
coroutineContext
then the main thread exits immediately. I think it’s something with parent child relationship.
l

louiscad

01/02/2023, 3:35 PM
You'd be using the context of the nearest launch, so the behavior should be exactly the same in the snippet you initially showed.
j

Jebus_Chris

01/02/2023, 3:39 PM
Oh got it, my bad. If this was instead a generic
suspend fun
instead of main do I have to create a new coroutine scope?
l

louiscad

01/02/2023, 8:36 PM
Yes, with
coroutineScope { }
5 Views