I'm trying to make a simple API service. The service is supposed to be connected to a Postgres database. Populating the database, requires me continuously parsing some files that contains lines of JSON data. The parsing would ideally happen by some trigger (e.g. a CLI command?) which then would execute some code. How can I do something like this, reusing database connections and models from my Ktor project?

Do you mean the CLI command uploads the json to be processed?

No, more like the CLI command (or whatever fits best) can be executed/triggered to parse any files found in a specified folder.

Ah, got it. And then inserts the data directly into the production database?

Exactly. I come from PHP/Laravel where would probably create an Artisan Command or a Worker and then have something like crontab or a message queue listener process my task. In this case, it would be processing the files, parse the JSON data and insert directly into the production database.

And where would the command run? In the production servers? (sorry, I'm not familiar with Laravel tooling).

Yes. In my very specific case, I would run Ktor application as an ECS Fargate service with a task. I would then use either a trigger or a scheduled task to execute the desired operation.

This may be quite different from the PHP world, though. When using PHP, you deploy your project's code directly into the server, and then you have access to all the tooling there, including Artisan Commands, correct? In the JVM world, you usually deploy a package (such as a jar), and the build tool is not there (your mileage may vary, of course).

Thank you for the explanation. I will try and dig a bit into the two options. Thank you for your time.

Anytime. Good luck. :-)

I have a bit of a background working with PHP/laravel, and this is one area where Java server apps differ significantly from PHP. The main reason is that each PHP request runs independently from the others, and there’s not really a concept of a “server process” there. But in Java servers such as Spring Boot or Ktor, the entire server is just a normal JVM process, so you can spawn background threads and store values in memory in a way that is impossible in PHP. So this makes it significantly simpler to do things like scheduling repeated jobs or enqueuing work that runs longer than a single request. Spring Boot has an

@Schedule

annotation to run repeated jobs on a custom schedule, which doesn’t need the kind of crontab+artisan CLI setup that you’d need in PHP. There’s also the Java stdlib Timer, which works similarly (and I believe the Spring Boot scheduler uses internally). These both just run in-process and don’t require much boilerplate or any additional infrastructure. Libraries like Quartz (or similar libraries) are good if you want the jobs to be resilient to server restarts or the tasks to be distributed/locked across the whole server fleet. But they can also be pretty large and complex to use, and often require additional infrastructure like databases or Redis caches that can make them a bit overkill for simpler tasks. Additionally, I’ve recently started experimenting with a new pure-Kotlin scheduling library that works on coroutines, and so might be feel more natural to use from Kotlin than Quartz or other java libraries. It’s still early in development and doesn’t even have snapshot builds available yet, but I’d love to get some feedback on it, because running Ballast Schedulers in Ktor is one of my main motivations for it! It’s meant to be a smaller, simpler library in the category of schedulers like Spring Boot and Timer, rather than the larger Quartz options.

Sorry this is a bit late. For what its worth: even though the quartz release has remained the same since 2019, I found it worked pretty well. It did need an additional database (I just set it up to use embedded SQLite). No additional processes were needed (no redis etc). It has ways to deal with missed jobs (e.g. if something should run every 24 hours, but at the scheduled time server is stopped etc). My experience with Quartz has been very positive. It has just sat there in my code doing its job without causing trouble for a long time for over 4 years.