Hi everyone! How can use krangl with Smile library...

# datascience

[Nusret Özateş]

Hi everyone! How can use krangl with Smile library? For example with Linear Regression (OLS in Smile)

[altavir]

I do not think that it is possible to do it out of the box. You need to convert data from krangl internal representation to smile internal representation. I am currently working on Table API that could allow to do it autimatically in future, but it is in early stage.

[altavir]

Also you can try to ping @holgerbrandl. Maybe he has some kind of solution for that.

[Nusret Özateş]

Okay thanks a lot! I am trying to do a machine learning project ( neural network project actually) this semester and I want to do it in Kotlin.

So I'm trying to find my tools that I can work without problem😊

[altavir]

The ecosystem is really youg. Please fill free to explain your problem and we will try to preorityze tools for it. I've already implemented auto-transformation of different matrix representations in kmath. I want to do the same for tables sooner or later.

[Nusret Özateş]

Well unfortunately I cannot find a research problem for now😂 But this semester I need to find a research problem that can be solved with Neural Networks and write a nice documentation/article about it.

And I like Kotlin a lot and want to add something to community with that project 😊 If you have an idea to do with Neural Networks I can work on it too!

[altavir]

What we really lack right now is the problem description. People coming from python usualy just say "do it like pandas/scipy/whatever". Kotlin actually could do better since it allows for more functional-style transformations so in order to do something, we need first to understand what needs to be done without references to existing Python solutions.

Typical example is the recurring theme of large in-memory files. You usually can't avoid them in Python since that is how NumPy works, but in kotlin we can do streaming API. I personally do not do machine learning so it hard for me to understand what is needed.

[altavir]

Also there are some ideas to unify the API of mathematics and data access libraries to add a Python-like experience where you can expect different libraries adhere to the same interfaces for data structures.

[Nusret Özateş]

For now I am looking for a workflow, for example in Python when you think about it

Pandas to get and play with data
Numpy to do mathematical/matrix operations
Matplotlib to show data as graph
Sklearn to do machine learning things like linear regression logistic regression etc
Maybe tensorflow for neural networks and machine learning ( not sure about it)

And they work with each other pretty well

For Kotlin

Pandas = krangl
Matplotlib = kravis
Sklearn and Tensorflow = Smile and Deeplearning4j and OjAlgo
Numpy = Nd4j

But for example Krangl dataframes cannot be smile dataframe, I need to convert it manually (I couldn't) or use Tablesaw library, but krangl works good with kravis since same person made them but I don't know how tablesaw and kravis will work with each other

[altavir]

I will add krangl bindings for DataForge-tables, and there are some bindings for ndarrays in kmath (ND4J is planned) but it is a bit more complicated in between. The problem is that pandas-like reader produces typed columns and it is not exactly the same thing as dense nd-arrays. You need to make a transformation of sorts and we need to think about the way tables are transformed into number arrays.

[Nusret Özateş]

You are right! Okay I will try to find my way after I learn more about neural networks, data science and after I decide my reaserch problem. It seems I can ask more detailed and helpful questions only after I decide what will be my project😂

[altavir]

Feel free to leave feature requests in kmath repository. I will separate table api from dataforge repo in near future and then we can discuss it more.

[Nusret Özateş]

Okay thanks a lot again! 😊

[Nusret Özateş]

Oh hey what will be the advantages of KMath over Nd4j?

[altavir]

Kmath is a wrapper. It has naive implementations for multiplatform, but basically, it delegates to platforms-specific libraries. ND4J is not implemented yet, but is planned: https://github.com/mipt-npm/kmath/issues/52

[altavir]

And it is possible in kmath to implicitely convert data from one library representation to another one... need to add an example about that.

[Nusret Özateş]

I get the idea and it seems awesome!

[altavir]

I proposed the same mechanics to Kotlin for DS team for other math tools and we are discussing it.

[jimn]

you can upload a bigquery table and run linear regression wtih about 15 minutes of RTFM, fwiw. it gets a little more involved if you want to upload a tensorflow model; and even though you are paying gcp for processed gigabytes, and not compute time, your time spent doing any kind of bigquery operations will be better spent offline in any local tool at all that you are familiar with, so you can get quick linear regression results with a well formed dataset you prep offline

[jimn]

I also showed up in the channel and asked about the same basic usecases, and wrote a tool in Kotlin that does what pandas does. I've spent no time at all dealing with smile, but if it is supported by a jackson adapter it's probably a few lines of code to set up a serialization solution to do some transforms.

[jimn]

i have come upon some time if you want to PM me. my wife is in Bali and I've ended my current engagement so I planneed to dedicate a few cycles to the tool.

[holgerbrandl]

@Nusret Özateş Indeed the krangl repo contains some example where we use smile to model data. Feel welcome to contact me via github or in case you have any questions or comments.