Hi there, what would be the best approach to run a...

# ktor

[frevib]

Hi there, what would be the best approach to run a Kafka consumer (or any message listener) next to Ktor? I know Ktor is a http library only, but most backend developers need to have some Kafka, MQTT or message consumer running next to their HTTP server.

For instance, is it recommended to run the Kafka consumer next to the Ktor server, like:

fun main() = runBlocking {
  launch(<http://Dispatchers.IO|Dispatchers.IO>) {
    embeddedServer(CIO, port = 8080, module = Application::modules)
    .start(wait = true)
  }

  launch(<http://Dispatchers.IO|Dispatchers.IO>) {
    startKafkaConsumer()
  }
}

or have the Kafka consumer be a module of Ktor, e.g.:

fun main() {
  embeddedServer(CIO, port = 8080, module = Application::modules)
    .start(wait = true)
}

fun Application.modules() {
  kafkaModule()
  ktorModules()
}

fun Application.kafkaModule() {
  launch(<http://Dispatchers.IO|Dispatchers.IO>) {
    startKafkaConsumer()
  }
}

Approach 1 seems more architecturally correct, having both Ktor server and Kafka consumer running in a coroutine on the same level. The 2nd approach makes Ktor managing the lifecycle of the Kafka consumer, which is arguably less good.

[Aleksei Tirman [JB]]

You can also have a parent coroutine scope, which could be canceled when the Ktor server or the Kafka consumer is canceled.

[frevib]

The two jobs from launch are in the same coroutine scope. We can wait for them to complete with them with joinAll(ktorJob, consumerJob) , and if one of the Jobs throws an exception, the whole scope is cancelled.

[frevib]

The Ktor server and Kafka consumer Jobs never complete btw, they are running as daemons, endlessly waiting for messages and HTTP requests.

[frevib]

Question remains: how do we do this, best practice way?