I have a `List<String>` with about 250 URLs....
# announcements
I have a
with about 250 URLs. Currently I am doing a
request to each url in a forEach loop so it’s synchronous. Obviously one of Kotlin’s amazing features is co-routines. Does anyone know of a good resource on how I could do this async? It’s currently taking 30 seconds while I feel like this could be done much much quicker async. Note I own all 250 URLs that are getting visited so I am not spamming anyone
kotlin documentation is a good way to learn about coroutines, if you are already familiar with concurrency and parallelizm
you can choose any library for working with network but i personally highly recommend using Ktor, here is the documentation how to start with the ktor-client https://ktor.io/docs/getting-started-ktor-client.html
Essentially it will look something like:
Copy code
val yourList: List<String> = ...
val results = yourList
  .map { scope.async { yourFunction(it) } }
The above links should explain where
etc comes from.
The thing that makes it parallel is not the map call, it's calling
multiple times before calling
careful with 250 URLs in parallel, it may be too much parallel requests (depends on how your http library is working. many have own limit on amount of connections per host)
limiting concurrency within coroutines is unfortunately not built-in, https://github.com/Kotlin/kotlinx.coroutines/issues/261
one simple workaround is
Copy code
val yourList: List<String> = ...
val results = yourList.asFlow()
    .flatMapMerge(concurrency = 8) { suspend { yourFunction(it) }.asFlow() }
although it doesn't preserve order
I don’t think that sliceable dispatcher is right solution for this case, it’s nice feature itself, but it will not limit concurrency, it will just limit amount of threads, which is not the case when you use asyncronous http client, you can run 250 parallel coroutines using 1 thread
solution with flow is good btw
I haven't used ktor client - is it fully asynchronous? http clients I've used are synchronous. but good point, a sliceable dispatcher would not limit asynchronous clients
yes, ktor is asyncronous as probably all modern http clients
I would argue, that if you use blocking API for an http client, you just need abstraction on top if it, rather to wrap it to coroutines every time. Also it usually much better just to use any http client which has built in request limit per instance
Though of course limit per use case is also important in some cases, when you don’t want to start too many requests from this particular code and prevent other requests to run
Also in this case to limit how many calls you could use a
from kotlinx.coroutines