Do you know any time series manipulation and modelling libraries for Kotlin?

Something could be done via kotlin-statistics. Also a lot of this in Java. But in general, you need to be more specific.

• Model and forecast time series using ARIMA with a sliding window cross-validation. • Resampling time series frequency. Ex.: seconds to minutes frequency or daily to monthly • Transform a list of values in sub lists of values depending on the time step

Have not heard of something kotlin-first, but I am sure there is something in Java world

Another thing, let's say I've a Java lib and want to use it in Kotlin while changing operators (doing operator overloading). Instead using X.add(Y), I use X + Y. Is it possible?

To be honest I think kotlin will be a great language for the execution part of algotrading. I won't use to develope the analytics and backtesting though. If you are absolutely desperate to use kotlin for that as well, I would choose a powerful computational platform like deeplearning4j and build on top of it everything. I would also avoid apache commons.

I’m using python to develop the forecasting models and for research (PhD). I’m building models for intraday ... Data is huge and I was not used to build models with this data size. I’m thinking to use Spark to data wrangling

Basic time-series manipulation does not require a lot of processing power. But not all tools are currently available in one place.

Krangl has some features for data manipulation but i think it is not suited for big data

Indeed, it is not. We are doing pretty big-data processing in Troitsk nu-mass analysis, which is close to time-serries, but it has a narrow focus and probably not good for other tasks.

@Filipe Duarte you should use https://code.kx.com/q/learn/ for intraday data

Thank you so much @Ролан! Do they give license for academics to use kdb? I thought spark fit time series data well. So in this case, for the sake of time management, it is best to follow your guidelines. Do you use Kotlin for high frequency financial data?

kdb 32 bit can be used by anyone, kdb 64 bit can be used for free on personal laptops with constraints on number of cores and RAM, for academics you get it for free with no constraints and you can run it on the cloud. I haven't seen anyone analysing market data in the JVM (execution yes of course, but that's different).

Do you use python api for kdb to do data analytics for market data?

I love

q

to be honest, but you can indeed work with

pyq

to call

q

from

python

, or vice-versa use

embedpy

to call

python

from

q

Do you talking about reading data from

kdb+

and store data using

h5

.. than using pytorch to load this data?

no

kdb

will be ways much faster than all this, I was talking about pure python/C++ here

but how to train the models with pytorch and then use the data with kdb+?

because you could use the reference of the array?

There is a client throught sockets

understand, and do you use Kotling for data analysis or something related?

Over the summer I plan to provide some bindings for pytorch to kotlin actually, that will add some of the data science tooling. My main motivation regarding doing so though is to bring powerful analytics tools closer to the data collection systems for doing realtime analytics. In trading typically what I have seen is that ETL and collections systems are running on the JVM. The data is then sent via some broker (like kafka, redis etc. or even just flushed to a database, or just via sockets) to yet a whole different service typically written in python doing the analytics. The result of those analyics is then sent to a client/decision/execution platform (again typically running on the JVM). Now, some simple analytics can happen on the JVM straightaway but that is limited, Spark can help but it has very poor perfomance. For any other numerically intensive modelling you need to go native.

So, for you, the ideal scenario would be to use the JVM Kotlin for real-time analytics? In high-frequency trading, do you think today the best way is to use python to prototype the models and strategies and C++ for production using Machine learning libs libtorch, tensorflow, mlpack I’ve seen some posts on HN about using JVM for HFT. Some firms are using it, but I don’t know much of the details. Maybe they operate in not so high-frequency.

@Filipe Duarte JVM/Kotlin are optimal for any use case, where you do not have a limit on memory consumption. The limitation right now is not in the language/platform, but in availability of libraries.

@altavir is biased 😀

I have experience in C++, Python and some in Julia. The thing is that if you need to do something once and never touch it again, it does not really matter which language to take. This is why academics tend to take C++ (for flexibility) and Python (for simplicity). But the problem is that in my experience there is no such thing as a program for a single use. There is always some kind of support involved. And for support you need good ecosystem. Python is considered good for prototyping and indeed even biased as I am, I am teaching python to physicists, but it all works well, while we do not want to turn those prototypes into the software.

I don't want to reinvent the wheel all the time tbh

Indeed, but when we do not algorithms, but full-scale software development, prototyping does not really work. You can't just translate something from python to, say C++ or Java without doing double work of rewriting and optimizing everything. Or you need to replicate familiar tools in every platform. Which you do now. My point is that right now the only advantage of Python ecosystem is the number of libraries existing there.

I am talking more about the maths. This is always very tricky to migrate. So I prefer to write that in C/C++ and keep it there. Then of course, because C++ is not so much fun for larger scale applications you integrate it into with other languages. JVM is certainly not the nicest club for mixing native but it's alright.

Indeed. The problem with that solution is that you need a highly qualified person like yourself to adopt that solution to specific needs. In science we usually do not have such people available. On the other hand, keeping record performance is not required either. Therefore we need something like python with good maintenance. In other words, Kotlin or Julia. My own choice of Kotlin over Julia has two reasons: much better tooling and support for industrial non-computing ecosystem (networking, data storage, etc).

Yep. And you can listen to some of Roman Elizarov't talks. He have experience with low-latency finance development in Java. C++ is probably irreplaceable in GameDev, where you need a low-level access to the hardware.

@altavir could you share links to some of these talks?

@Filipe Duarte I recommend you to use the resources available on the

kdb

website. That stack was specifically designed to help people solve the problems you are working on.

I’m going to dive deep