What’s the best or proper way to use coroutines with api and database calls? The api is using ktor c...

# coroutines

[Jebus_Chris]

What’s the best or proper way to use coroutines with api and database calls? The api is using ktor client (it can suspend) and the db is using jdbc (blocking) with sqldelight+hikari.
Say I need 1000 api calls and then 1000 db inserts, right now I just make 1000 coroutines for the api and then 1000 for the db as the api calls return (all using Dispatchers IO). Right now main handles all the coroutines. Should they instead be in the api/db classes themselves or even in both spots? This is just a script, but would the approach change for a web app in ktor?

Simplified code below:

fun main() {
    runBlocking { 
        launch {
            values.forEach {
                CoroutineScope(coroutineContext + <http://Dispatchers.IO|Dispatchers.IO>).launch {
                    val data = api.getData(it)
                    channel.send(data)
                }
            }
            for (i in 1..values.size) {
                val data = channel.receive()
                CoroutineScope(coroutineContext + <http://Dispatchers.IO|Dispatchers.IO>).launch {
                    db.insert(data)
                }
            }
        }.join()
    }
}

[jw]

I would use a single coroutine for the database and a channel to send the items to insert

[Jebus_Chris]

Wouldn’t that make the database calls execute sequentially?

[jw]

Yes

[jw]

Write locks are exclusive so even with 1000 coroutines and 1000 threads executing perfectly in parallel it will be sequential

[jw]

So you might as well have only a single coroutine and eliminate lock contention

[Jebus_Chris]

I see, so if they were reads then it would make sense.

[Jebus_Chris]

Might be more of a design questions, but should the coroutines be in the main class (or calling class/function) or should they instead be in the api or the db class?

[Colton Idle]

Write locks are exclusive so even with 1000 coroutines and 1000 threads executing perfectly in parallel it will be sequential

wait. maybe dumb question. but is that basically true of all SQL* databases? I never knew that it'd be sequential without anyway to write it. How do backend DBs keep up with tons of writes? 🤯

[jw]

Depends on the DB, but generally you write to a log file which is append only

[jw]

Then that's applied asynchronously

[Colton Idle]

I should probably read up on how databases work at a fundamental level, but i was reading a topic similar to this which is funny that this came up in kotlinlang slack. as always thanks for teaching.

[jw]

They're a great source of learning because performance and correctness are both critical

[louiscad]

You can replace CoroutineScope(coroutineContext + <http://Dispatchers.IO|Dispatchers.IO>).launch by just launch(<http://Dispatchers.IO|Dispatchers.IO>) FYI.

[Jebus_Chris]

I remember messing around with that. If I don’t have coroutineContext then the main thread exits immediately. I think it’s something with parent child relationship.

[louiscad]

You'd be using the context of the nearest launch, so the behavior should be exactly the same in the snippet you initially showed.

[Jebus_Chris]

Oh got it, my bad. If this was instead a generic suspend fun instead of main do I have to create a new coroutine scope?

[louiscad]

Yes, with coroutineScope { }