Hey everyone, I was playing with Kafka, and looking around I didn’t really find any extension on top...

# server

[simon.vergauwen]

Hey everyone,
I was playing with Kafka, and looking around I didn’t really find any extension on top of the Kafka SDK for example to write Flow based Kafka programs.
Was wondering if there would be any interest in such a library, and/or if people are interested in contributing. Or if there is an existing library already out there. I have some operators, and examples ready if someone is interested.

https://github.com/nomisRev/kotlin-kafka

[Jacob]

Does this work with manual commit mode for the consumer? What sort of acknowledgement are you getting when producing the messages, or is it just fire and forget?

[Jacob]

I hear the words "playing with kafka" and My head shouts "I wonder if he's aware of all the gotchas involved with kafka's slightly unusual messaging model"

[hfhbd]

I also created a Kafka wrapper using Flows, but only for internal usage without all Kafka functions (and there are many!). Ideally, instead using existing Java API, rewritten in from scratch in Kotlin common code would be incredible, by using the native network api.

[simon.vergauwen]

@hfhbd Yes, that would be incredible but would require quite a lot more of time/work.

[simon.vergauwen]

@Jacob I’m aware of some of the gotchas, the idea is not abstract things away but just provide. I’m reading through the Kafka definitive guide while writing this.

The goal would mostly be to only expose some sugar syntax methods where needed but not rewrite or wrap the existing SDK.

[simon.vergauwen]

I was a bit surprised there was nothing out there, a popular option is probably also kafka-reactor which you can also use with KotlinX Flow throught the integration module

[Jamal Khan]

If you willing to use spring, spring-cloud-stream with the Kafka Binder and kotlinx-coroutines-reactive should do the trick (convert a Flux to a Flow using the asFlow extension)
This link should create a starter for you: https://start.spring.io/#!type=gradle-project&language=kotlin&platformVersion=2.6.1&p[…]cessor&dependencies=web,cloud-stream,kafka,actuator,devtools
and then you can add the kotlinx deps