I'm trying to make a simple API service. The serv...

# ktor

[Dan Storm]

I'm trying to make a simple API service. The service is supposed to be connected to a Postgres database.
Populating the database, requires me continuously parsing some files that contains lines of JSON data.
The parsing would ideally happen by some trigger (e.g. a CLI command?) which then would execute some code.

How can I do something like this, reusing database connections and models from my Ktor project?

[marcospereira]

Do you mean the CLI command uploads the json to be processed?

[Dan Storm]

No, more like the CLI command (or whatever fits best) can be executed/triggered to parse any files found in a specified folder.

[marcospereira]

Ah, got it. And then inserts the data directly into the production database?

[Dan Storm]

Exactly. I come from PHP/Laravel where would probably create an Artisan Command or a Worker and then have something like crontab or a message queue listener process my task.

In this case, it would be processing the files, parse the JSON data and insert directly into the production database.

[marcospereira]

And where would the command run? In the production servers? (sorry, I'm not familiar with Laravel tooling).

[Dan Storm]

Yes. In my very specific case, I would run Ktor application as an ECS Fargate service with a task. I would then use either a trigger or a scheduled task to execute the desired operation.

[Dan Storm]

And that would be within the same environment.

[marcospereira]

This may be quite different from the PHP world, though. When using PHP, you deploy your project's code directly into the server, and then you have access to all the tooling there, including Artisan Commands, correct?

In the JVM world, you usually deploy a package (such as a jar), and the build tool is not there (your mileage may vary, of course).

[marcospereira]

The options I see are:

[marcospereira]

1. Create/deploy another jar that you can run in the prod server, and it will have access to the database. It would work like java -jar my-command.jar.

[marcospereira]

2. Add an endpoint to your ktor application that triggers the process, and then just call that endpoint. The endpoint would be more like "hey, there are new files to import in that files-to-process-dir". The endpoint can then use Quartz to schedule the import task.

[marcospereira]

Check this out: https://www.marcogomiero.com/posts/2022/ktor-jobs-quartz/

[marcospereira]

One advantage here is that it would even use the same connection pool as the ktor application, not only the same db credentials.

[Dan Storm]

Thank you for the explanation. I will try and dig a bit into the two options. Thank you for your time.

[marcospereira]

Anytime. Good luck. :-)

[Casey Brooks]

I have a bit of a background working with PHP/laravel, and this is one area where Java server apps differ significantly from PHP. The main reason is that each PHP request runs independently from the others, and there’s not really a concept of a “server process” there. But in Java servers such as Spring Boot or Ktor, the entire server is just a normal JVM process, so you can spawn background threads and store values in memory in a way that is impossible in PHP.

So this makes it significantly simpler to do things like scheduling repeated jobs or enqueuing work that runs longer than a single request. Spring Boot has an @Schedule annotation to run repeated jobs on a custom schedule, which doesn’t need the kind of crontab+artisan CLI setup that you’d need in PHP. There’s also the Java stdlib Timer, which works similarly (and I believe the Spring Boot scheduler uses internally). These both just run in-process and don’t require much boilerplate or any additional infrastructure.

Libraries like Quartz (or similar libraries) are good if you want the jobs to be resilient to server restarts or the tasks to be distributed/locked across the whole server fleet. But they can also be pretty large and complex to use, and often require additional infrastructure like databases or Redis caches that can make them a bit overkill for simpler tasks.

Additionally, I’ve recently started experimenting with a new pure-Kotlin scheduling library that works on coroutines, and so might be feel more natural to use from Kotlin than Quartz or other java libraries. It’s still early in development and doesn’t even have snapshot builds available yet, but I’d love to get some feedback on it, because running Ballast Schedulers in Ktor is one of my main motivations for it! It’s meant to be a smaller, simpler library in the category of schedulers like Spring Boot and Timer, rather than the larger Quartz options.

[Mike Dawson]

Sorry this is a bit late. For what its worth: even though the quartz release has remained the same since 2019, I found it worked pretty well. It did need an additional database (I just set it up to use embedded SQLite). No additional processes were needed (no redis etc).

It has ways to deal with missed jobs (e.g. if something should run every 24 hours, but at the scheduled time server is stopped etc).

My experience with Quartz has been very positive. It has just sat there in my code doing its job without causing trouble for a long time for over 4 years.