Yeah, to me it’s just interesting that things that I process without any conscious effort (language, vision) are best solved by the black box of a neural network. It’ll be interesting to see how the utilization of NNs continues to evolve since we have only had practically useful neural networks for a few years. It’s been interesting to see NNs applied to video game AI as well.

Hello guys, yesterday we had a talk with @elizarov about development of scientific/mathematics libraries for kotlin and he recommended to start a discussion here since we do not have dedicated channel for that. My team is working on a bunch of scientific libraries (the application is mostly physics, but they should be universal). Currently we are working on a basic mathematics (https://github.com/altavir/kmath/tree/dev). I personally think that Kotlin has a great future as a language for science so, it would be good to create a robust scientific community around. So, speaking about math, in

kmath

we use rather experimental approach using unique context oriented kotlin features rather than copying details from python. I studied https://github.com/kyonifer/koma and there are a lot of good things there and a lot of work done. I think, that we need that library or something similar for people coming form Python, but sadly it does not suite more advanced mathematics use that we need. Python style is probably good for data science, but the type system is too crude do something complicated. My proposition is to devise a common mathematics interfaces and specifications for kotlin/multiplatform and then ensure that all implementations follow those interfaces. The same goes for plotting (we are currently working on plotting interfaces similar to matplotlib in functionality, but with modern declarative api).

i think there's a huge advantage to having something familiar to the python/matlab communities, but backend implementations are something the user hopefully doesnt interact with often

As for performance of boxing, I think I found a remedy. You can look here: https://github.com/altavir/kmath/tree/dev/kmath-core/src/jvmMain/kotlin/scientifik/kmath/structures documentation is thin yet. But preliminary tests show no performance loss against non-boxing code.

Hi, I would like to join the discussion on building Kotlin's mathematical libraries by showing you what I've done. I'm currently in a Signal and Image Processing Master where we are continuously assigned projects in Python and Matlab as they are the standard in the industry for quick prototyping and experimentation. Having experience with Kotlin, I see a huge opportunity to improve the existing tools to take advantage of a modern programming language and it's excellent tooling, the same way you do. I've been using @kyonifer's koma, which is very helpful and familiar to newcomers from Python and I had the same thoughts as @altavir, mimicking Python's and/or Matlab's syntax leads to the same drawbacks I was trying to avoid. I'm very interested in @altavir's kmath and his context driven approach, I think it is modelling mathematical primitives the right way. I believe the reason why Python became the first choice for data science is because of numpy. Python's data science community managed to converge on a common library for numerical primitives which, unlike Matlab or other's, it's based on ND Arrays, not matrices. ND Arrays are more natural to model problems in engineering, particularly for modern applications such as differentiable models (a.k.a "neural networks/AI") and the fact that all engineering libraries use the same numerical primitives on their interfaces make's it easy to compose and share work. This is the reason why I started my own project for NDArrays on pure Kotlin: https://github.com/TomasVolker/numeriko/tree/master/numeriko-core to see how such a library would look like. It is in a completely experimental state, far from being fully documented and tested and my intention is not to fragment even more the library ecosystem but to show to you how I envision such a library that takes advantage of Kotlin's features and uses the same conventions. Equally important, visualization is fundamental for data science and Python also has matplotlib for this. I have a basic matplotlib wrapper implemented in Kotlin that uses DSL for a declarative syntax: https://github.com/TomasVolker/numeriko/tree/master/kyplot. This should be a temporary solution for a more complete and powerful Kotlin library based on OpenGL, which I'm already experimenting with for 3D plots. There is a lot to discuss about implementation details, but what I believe is a priority is to agree on how we want a common ND Array library on which we all should build on, an idiomatic Kotlin numpy equivalent. I would like to know your views/thoughts/feedback on the next steps for building Kotlin's datascience, engineering and math libraries.

You are going to end up implementing a Prolog-like in Kotlin @thomasnield 😄

@altavir 🤔 okay I see where you are going with this. But what if my business problem says "we get x customers per 6 minutes" and "we process each customer in y minutes"? That is pure Poisson right there and discrete by nature

Lets' move then

Trying to figure out ecosystem of Kotlin-Statistics and how I will make 2.0 multiplatform, and possibly use KMath for the linear algebra stuff....

Is there a clean way to print the matrix built in yet?

@altavir I might be naive, but are there any thoughts regarding BigDecimal support or if it is even necessary? I know it's a JVM thing and can get marginalized by having operations delegated to Double anyway...

so given the leaps in kotlin scripting.. i think it would be possible (once a few more bugs are fixed) to make a notebook for kotlin note that i know barely anything about notebooks but the main point is to load datasets, transform them over multiple steps and easily visualize data right ?

Interesting read on Swift for data science: https://www.fast.ai/2019/01/10/swift-numerics/

@altavir if you have a Twitter account, feel free to engage and see if you can influence Jeremy ☝️

Anyone used this? https://github.com/JetBrains-Research/viktor

I’m absolute zero in statistics, but I need to report couple of values (mean, error, confidence interval, etc) out of the sample dataset (DoubleArray). It is all in Kotlin Multiplatform (actually a benchmarking suite), so I can’t use any JVM library. I would also like to avoid dependencies at all if possible. Can someone point me to a minimum common Kotlin code I can steal shamelessly (Apache 2)? Or help me develop one. I’ve got this from some place (I copied it several times from a project to project, so I don’t remember where it came from originally) https://github.com/orangy/gradle-benchmarks/blob/master/runtime/nativeMain/src/org/jetbrains/gradle/benchmarks/NativeBenchmarksStatistics.kt Looking at some resources, I see TDistribution, inverseCumulativeProbability and such, but unfortunately I don’t have any idea what is it. The target goal is to generate JMH-like JSON (sample in a thread)

@orangy will It be possible to create configurations for individual tests in the plugin? I really miss it in current JMH plugin.

Guys, a question. What big data is big enough for you. I plan to re-implement some descriptive statistics routines for kotlin-statistics and it is important to know what data sample sizes are used. For example if data is of TB size, one should go for different algorithm (there is an ongoing discussion in commons-math email list).

@hudsonb yes, as I updated my article and put it on Medium I realized we needed to apply that to bin operators. If you can put in a PR that would be awesome, and I can update the blog article.

Is anyone currently using spark-sql 2.4 with Kotlin? I'm running into a weird bug that I'm not sure if anyone has been able to figure out. When importing that dependency my code get's a class not found error but works fine from running it in IntelliJ. When opening the Jar I can see it in the class path so stumped on what the issue is. Even weirder is I have a file that works perfectly fine with spark-core and the RDD api, however it breaks with the same error when I import the spark-sql library. As far as I understand that library is dependent on core and should be packaging it as well.

My last article for awhile: https://towardsdatascience.com/how-it-feels-to-learn-data-science-in-2019-6ee688498029

@Nikky don't become like this guy though. https://www.forbes.com/sites/janetwburns/2016/10/13/elon-musk-and-friends-are-spending-millions-to-break-out-of-the-matrix/#42c3d345ce17

Hello, is there a good Machine Learning library for Kotlin besides Tensorflow? I want to implement a type of Recursive Neural Network

Going along with the topic of machine learning/ai/deep learning/neural networks I'll like to share some experimental projects I've been working on. As I mentioned some time ago, I've been experimenting with Tensorflow and Kotlin. Now that gradients are being ported to C++, they are available on other backends apart from Python and I see a lot of potential in Kotlin to develop complex models using DSLs. This is the motivation of Komputo (https://github.com/TomasVolker/komputo), a Kotlin wrapper around Tensorflow for Java. It is in very early stages but I was able to build and train fully connected and convolutional models all in Kotlin using GPU acceleration, you can check the MNIST examples on the repository. I was also able to load the Yolov3 object recognition model with Komputo, here is the repo: https://github.com/TomasVolker/kolo. I'm using Numeriko (https://github.com/TomasVolker/numeriko) for ndarray manipulation, I've been putting some effort on it lately to improve it's performance. It's clear that to enable this kind of applications we need an idiomatic, efficient and complete ndarray library, so I'm working on it whenever I have some free time. I see Numeriko as a lower level library than @altavir's kmath, we can eventually create contexts to make them compatible. Let me know if any of you are interested in contributing.

is there some kind of standard realife task that is used fow notebooks ? i am writing something very similar that can process data in multiple steps and visualite it now it would be useful to copy some sort of sample usecase from somewhere that can be done using jvm libraries

Well, JNI is not about build. You have to make a wrapper, then compile both parts. It is not pleasant. One can use JNA or JNR instead

Do you have a link on this? It's a subject for data2viz as we'll probably use a lot of matrix operations for the composition of transformations.

I’m considering Kotlin for production. Does someone know how CoreNLP compare against Spacy and Smile against Sklearn ? i.e are these viable options ?