Happy new year! I want to make a Kotlin Native app that will capture some text messages and then fo...

# kotlin-native

[Adam S]

Happy new year!

I want to make a Kotlin Native app that will capture some text messages and then forward them to a server. The messages need to be ordered, and I’m targeting Windows/Mac/Linux.

Can anyone recommend a persistent file-based FIFO queue, or advice on implementing one? I’m trying with SQLDelight, but I wondered if there are alternatives.

[kevin.cianfarini]

Don't make the order implicit. Make it explicit by adding a time column in SQLDelight. Then just track the processed/unprocessed state as it's handed off to the server. No need for a queuing system if you do it that way.

[Michael Paus]

I think you should first clarify your requirements. Why do you need a persistent file-based FIFO queue if you want to do is send the text messages to a server.

[Saku Ytti]

Depending on wall clock being monotonically increasing is not reliable, and you shouldnt use it for ordering or measuring duration.

[Adam S]

sure - is something in particular unclear?

In short I’m making a Factorio mod that forwards data to a server, but because Factorio mods are very restricted they can’t access the internet. So the mod is communicating via stdout (I might change to log files, performance dependent).

I tried using the Docker syslog log driver, but it drops too many messages when the syslog server and container aren’t on the same machine.

So I want to make a K/N app that captures messages from another process or Docker container (either via stdout, or as a Syslog server or potentially files), and then forwards them to a server, or stores them until the server is available.

The messages already contain a timestamp, so I don’t need another. If the K/N app can run locally then it will get practically all messages in order, and I can drop any out-of-order messages server-side.

[Saku Ytti]

This implies you regularly see them being out of order, why is this? I imagine perhaps naively loop that reads this ‘text message’ from somewhere then writes out to network as UDP or TCP packet. And this naive model does not cause reordering.

[Adam S]

the ordering has been okay, I think the problems with the Factorio Server->Docker Syslog->my Syslog server chain was that the Docker syslog server couldn’t handle the larger messages, so those messages never arrived.

[Adam S]

I didn’t look too much into it because using syslog works for local development, but it’s too restrictive/technical for the mod to be easy to use.

[hfhbd]

I don't know the limits of factorio modding but if you are able to write to files, what about file sockets?

[Adam S]

I haven’t tried, I would be surprised if it was allowed

[Adam S]

mods are written in Lua, and the API is very restricted to ensure good performance and synchronisation between multiplayer clients

[Adam S]

capturing the messages via stdout is working okay, it’s just sending them to my server over the internet, that’s flaky