Question about coroutines and if what I’m doing is...

# announcements

[galex]

Question about coroutines and if what I’m doing is counter-productive:
I’ve got 111 urls to download and used coroutines with async {} to do it in parallel.
By using CommonPool it takes 10 seconds but by using async(newSingleThreadContext()) per url I get to 6 seconds, almost twice as fast!
It’s a gradle plugin meaning a few seconds after downloading all the data the program finishes so I am not worried about the number of threads.
Does this makes sense to do it the way I did or is there a better way to be faster while using less threads?

[Czar]

I've found that it makes sense to test on your specific workload. Generally it does not make sense to have more threads then you have processor cores, because of the context switching, but! if your workload is not CPU intensive (does not load processor core to 80-90%, then sometimes it makes sense to have more threads) YMMV, that's why testing several approaches is the best approach if you really need that optimization.

[groostav]

this is very interesting, @galex, I think to rephrase with czar said, by using many single-threads instead of nCPU's worth of threads, you basically shift the workload on to the net stack. In this sense, 111 threads is more efficient than 4, since you can now tell your networking stack everything that you want to download at the beginning instead of telling it that you want file E only after file A is finished downloading

[groostav]

I would still think that a newCachedThreadPool() makes a bit more sense though, and in either solution you're going to have a pretty large amount of overhead in simply assigning enough memory for the stacks of all of those threads. though that overhead is likely to be a drop in the bucket compared to the 200ms+ network latency

[galex]

Thank you for the explanation!

[xap4o]

Are you using blocking network calls or nio?

[galex]

blocking network