A modern programming language that makes developers happier.

kotlinlang

I tried hooking up arrow fx in kafka consumer. Atleast for the use cases where order is needed parTraverse didn’t make sense. Because it will take the whole batch and process all the records in the batch concurrently. The alternative can be queue. But then kafka consumer client already comes polling apis. So why put an extra layer on it. Then there is another case where you want to commit the batch only if all the records are processed successfully or you want to commit each record. 
Bottom line was that I faced problems with the offsets and couldn’t rely on the order of records when I process them concurrently which is the usp of using kafka. 
Kafka consumer client already comes with asynchronous apis for when you don’t have to worry about offset management. 
So I basically run a good old synchronous while loop and process each record with Arrow apis and call .suspended() in the end to make it blocking/synchronous 