I have a `List<String>` with about 250 URLs. Currently I am doing a `GET` request to each url ...

# announcements

[Jilles Soeters]

I have a List<String> with about 250 URLs. Currently I am doing a GET request to each url in a forEach loop so it’s synchronous. Obviously one of Kotlin’s amazing features is co-routines. Does anyone know of a good resource on how I could do this async? It’s currently taking 30 seconds while I feel like this could be done much much quicker async. Note I own all 250 URLs that are getting visited so I am not spamming anyone

[Arslan Armanuly]

https://play.kotlinlang.org/hands-on/Introduction%20to%20Coroutines%20and%20Channels/01_Introduction

[Arslan Armanuly]

https://kotlinlang.org/docs/coroutines-guide.html

[Arslan Armanuly]

kotlin documentation is a good way to learn about coroutines, if you are already familiar with concurrency and parallelizm

[Arslan Armanuly]

you can choose any library for working with network but i personally highly recommend using Ktor, here is the documentation how to start with the ktor-client https://ktor.io/docs/getting-started-ktor-client.html

[CLOVIS]

Essentially it will look something like:

val yourList: List<String> = ...
val results = yourList
  .map { scope.async { yourFunction(it) } }
  .awaitAll()

The above links should explain where scope etc comes from.

[CLOVIS]

The thing that makes it parallel is not the map call, it's calling async multiple times before calling await.

[gildor]

careful with 250 URLs in parallel, it may be too much parallel requests (depends on how your http library is working. many have own limit on amount of connections per host)

[ephemient]

limiting concurrency within coroutines is unfortunately not built-in, https://github.com/Kotlin/kotlinx.coroutines/issues/261

[ephemient]

one simple workaround is

val yourList: List<String> = ...
val results = yourList.asFlow()
    .flatMapMerge(concurrency = 8) { suspend { yourFunction(it) }.asFlow() }
    .toList()

although it doesn't preserve order

[gildor]

I don’t think that sliceable dispatcher is right solution for this case, it’s nice feature itself, but it will not limit concurrency, it will just limit amount of threads, which is not the case when you use asyncronous http client, you can run 250 parallel coroutines using 1 thread

[gildor]

solution with flow is good btw

[ephemient]

I haven't used ktor client - is it fully asynchronous? http clients I've used are synchronous. but good point, a sliceable dispatcher would not limit asynchronous clients

[gildor]

yes, ktor is asyncronous as probably all modern http clients

[gildor]

I would argue, that if you use blocking API for an http client, you just need abstraction on top if it, rather to wrap it to coroutines every time.

Also it usually much better just to use any http client which has built in request limit per instance

[gildor]

Though of course limit per use case is also important in some cases, when you don’t want to start too many requests from this particular code and prevent other requests to run

[Youssef Shoaib [MOD]]

Also in this case to limit how many calls you could use a Semaphore from kotlinx.coroutines