- We want at least one package to follow Python co...

# science

[altavir]

- We want at least one package to follow Python conventions so it will be easy for Python guys to switch to Kotlin. It probably should still follow common API somewhere on lower level.

[kyonifer]

If the answer to the above is yes, then in the short term Koma could simply add a backend for the lower level kotlin numerical library that is being built. As our homegrown lib becomes feature complete, koma could start to deprecate support for other backends as they become less relevant, or even eventually stop supporting other backends if our pure kotlin lib has competitive performance/capability/platform support versus using platform libs.

[altavir]

I think that this is the place Koma is suited best. We need something python-like and koma already does that. Though I do not really like how it looks like on the inside. So I think that we keep designing a set of APIs to suite everyone needs (with your input of course) and when it is ready, make koma implement it so we could exchange basic mathematics objects between different libraries (not thinking about performance).

[bjonnh]

What do you mean by "not thinking about performance"? ç

[altavir]

I do not know how it will work in the end, but in my opinion modules focused on specific features + common API is better than universal monolithic library.

[altavir]

I mean that you need to think about performance when you are doing some things inside the library, but when you are moving an object form one implementation to another one, performance is not important since it is quite rare.

[bjonnh]

I agree with the Python-like conventions (numpy-like more exactly). It would be nice to get these to work with more statistical/modelling packages (similar to scipy) and dataframes (similar to pandas)

[kyonifer]

@bjonnh so far we have krangl for dataframes and kotlin-statistics. Both authors are on slack @thomasnield @holgerbrandl

[bjonnh]

I am aware of that, but that would be nice if all those things were handling the same low-level representations (the same as pandas and scipy that use numpy in the back)

[kyonifer]

yep, i meant i could ping them here to weigh in

[altavir]

Yep. Now we need common app to make them talk to each other and focus on their own features instead of duplicating them.

[Zelenyi]

Koma is similar to numpy, but no enough, I think,, we need in more usefull and more python-like scientifik DSL.

[bjonnh]

We may also want to find out what numpy's design defects are (that they can't get rid of because of backward compatibility) and avoid having the same problems

[Percha]

@kyonifer I agree with koma's goal, it can provide a familiar API for newcomers from python and matlab. It's important to start discussing about how we want the lower level API, I'll start working on a KEEP like proposal based on numeriko to have on @altavir KEEP-math repository.

[altavir]

Here comes pythonic guy. @Zelenyi please start with writing a proposal for Keep-math about what you really need and why do you think that Python approach is good. We can discuss it later in our slack.

[kyonifer]

@Mikhail the numpy-like API can still be extended and fleshed out for those that prefer it (I'll admit I'm one of those!). Supporting offloading the work to a low level kotlin lib doesn't get in the way of koma growing. I'd be interested in you opening issues for koma if you have ideas on how it could be better

[kyonifer]

wrong user tag apparently: @Zelenyi

[altavir]

@bjonnh numpy limitation come mostly not from backward compatibility, but from limitations if underlying Fortran libs and performance issues. For example you can't use functions, they would be super slow.

[bjonnh]

is there a way to avoid that for us?

[kyonifer]

openjdk's jit is much better than cpython in terms of performance. Where the python guys have to drop to cython, numba, numexpr etc. to get more performance, we could theoretically drop to kotlin/native to yield even better performance (once optimization passes have been done on k/native, its not ready yet). So I think kotlin is already better poised to not have the same issues, as long as we design carefully for performance along the way.

[kyonifer]

my biggest beef with numpy is really python. Among other things, its lack of static typing. it makes it really hard to use for larger projects in technical computing with everything being dynamic and failing at runtime. I would like anything we come up with to be statically typed and avoid runtime casting as much as possible, as this is one strong advantage of using kotlin.

[altavir]

I do not think that native performance will be ever better, but functions in kotlin are very cheap compared to pyhon. Also we do not need to think about passing arrays to Fortran. I've currently implemented lazy operations which are not possible in Python.

[kyonifer]

the advantage of k/native is that you can call directly into vectorized (SSE optimized) assembly when you need to. the JNI/JNA add a good deal of overhead so one has to be careful how often they go through the JNI barrier (do loops in C), and the jvm currently has issues with generating SSE jit output, and compiled code will never beat hand-rolled assembly like Intel MKL anyway. I don't think code written in k/native will ever be faster than k/jvm (in fact probably slower without the awesome optimizer that is hotspot), but easy and low-latency access to C ABI is useful

[thomasnield]

I am not an expert by any means in developing a computational library at the lowest level. However I do think keeping the core Library as minimalist and

[thomasnield]

Unopinionated as possible should be the goal. We are probably never going to agree on the different levels of abstractions that will be built. And those abstractions should contain the risks of design mistakes, not the low level BLAS.

[altavir]

Indeed, I think everyone will agree with that. Those are the details of implementation and should be hidden.

[altavir]

By low level I usually mean abstract which is probably high level by general conventions.

[thomasnield]

I do think we should keep the core statically typed as much as possible. Developers building the abstractions will hate having to cast things. I think ojalgo maximizes the use of static types pretty effectively on the java platform. I have also seen terrible Java libraries that look like they were written by a python developer who spent more time fighting static typing than leveraging it.

[altavir]

You need to remember that kotlin sometimes allows to use generics much more effectively than Java. For example to avoid boxing. So static typing it is, but probably a little bit more generic. I have to go to sleep right now, but I am looking forward to continue discussion tomorrow. Also remember that pragmatic development starts with use-cases. So feel free to contribute them.

[Percha]

We all agree on building the core taking advantage of the static typing as much as possible, it is the main reason I want to use Kotlin instead of Python. For this reason, I believe we shouldn't try to copy numpy's API, it is thought for a dynamically typed language and breaks with Kotlin principles and idioms.

[thomasnield]

@altavir haha I wonder if this initiative will create KEEP proposals finding new and creative ways to inline primitives and functions...

[Zelenyi]

@thomasnield Nobody don't request to fight with statically typed, but it muste be hidden. If in code contains heap of <Int>, this is ugly. I cann't says about java-library which was created by python-developer, but scientifik java library from java-developers also it not sugar.

[Percha]

And for the performance issues, I'm not sure if you are familiar with this repository: https://github.com/JetBrains-Research/viktor it uses JNI to call SIMD instructions when available. It is from Jetbrains Research, I don't know if any of you have been in touch with them and if they are interested in the discussion.

[altavir]

They are.