https://kotlinlang.org logo
#coroutines
Title
# coroutines
d

df

11/18/2020, 8:40 PM
hey there. I have a list of a few million urls and i want to fetch the head response for them. What would be the best approach if I want to limit the amount of urls that can be fetched concurrently? Of course I could have a sequence that produces the requests, chunk them and process one chunk at a time but this feels like my default coming-from-a-java-world solution instead of something tailored to kotlin / around co-routines.
j

jw

11/18/2020, 8:52 PM
start a coroutine which iterates the list and dumps them into a channel. start N coroutines which take from the channel and perform the HEAD request. N is your concurrency number.
d

df

11/18/2020, 8:54 PM
sorry I only come up with this now but this solution wouldn't allow me to stop the processing as soon as one request fails right?
But I like the solution. I'll just give it a try :)
b

bezrukov

11/18/2020, 9:13 PM
It will if you want to. Just do not catch exceptions, so as soon as one request fails it will cancel the whole scope
7 Views