If I do a partraverse, what would be the correct way to limit the number of elements that gets execu...

# arrow

[bjonnh]

If I do a partraverse, what would be the correct way to limit the number of elements that gets executed in parallel?

[rt]

idk what is arrow's semantics but if it's anything close to monix/cats io, this should work: https://monix.io/docs/2x/tutorials/parallelism.html#imposing-a-parallelism-limit

[stojan]

You can use a semaphore https://arrow-kt.io/docs/0.10/apidocs/arrow-fx/arrow.fx/-semaphore/index.html

[bjonnh]

so I just make one of these and do a

[bjonnh]

semaphore.flatMap { it.acquire()}
myFunctionthatneedstorunonlyNtimesMaxinParallel()
semaphore.flatMap { it.release() }

[stojan]

Or use withPermit

<https://arrow-kt.io/docs/0.10/apidocs/arrow-fx/arrow.fx/-semaphore/with-permit.html#withpermit|https://arrow-kt.io/docs/0.10/apidocs/arrow-fx/arrow.fx/-semaphore/with-permit.html#withpermit>

[bjonnh]

neat

[bjonnh]

I'm still probably not doing things the right way, I'm calling ktor client from there and if I have too many requests, I get timeout errors

[bjonnh]

I can have 5 concurrent requests with 1000 queries in my list for partraverse but if I have 10000 it start failing

[raulraja]

in the new fx coroutines lib there is partTraverseN that takes an arg to limit parallelism

[bjonnh]

!

[bjonnh]

is that in 0.10.5?

[bjonnh]

looks like it is 0.11.0-snapshot

[bjonnh]

I tried to load it but now IO.fx doesn't work anymore

[raulraja]

arrow-fx suspend has the IO api in suspended style

[raulraja]

IO is going away because it’s much slower and fx-coroutines can do the same with easier syntax and same guarantees

[raulraja]

https://github.com/arrow-kt/arrow-fx/blob/master/arrow-fx-coroutines/src/main/kotlin/arrow/fx/coroutines/ParTraverse.kt#L53

[raulraja]

it just relies on Semaphore if you just want an add-hoc version of it before migrating IO