https://kotlinlang.org logo
Join Slack
Powered by
is there some kind of standard realife task that i...
# datascience
n
is there some kind of standard realife task that is used fow notebooks ? i am writing something very similar that can process data in multiple steps and visualite it now it would be useful to copy some sort of sample usecase from somewhere that can be done using jvm libraries
a
Not sure I understand the question
There is a beakerx jupyter extension and it in dire need of kotlin evaluator replacement
Here is the issue https://github.com/twosigma/beakerx/issues/7894. Help there would be very welcome. I intended to do it myself, but I'am currently very short on time.
n
well i started making my own completely seperate from jupyter, @amanda.hinchman-dominguez is doing the tornadofx vizualization we are just not happy with the need to run a webserver to run kotlin snippets
a
Jupyter has its limitations, but everyone use it and I am not sure it is possible to create a standalone visualization framework with comparable capabilities. Beakerx already has some visualization capabilities for JVM languages and, what is much more important, autotranslation feature to move data between python, and, say, kotlin. It seems reasonable to invest in it instead of creating something completely new.
n
also a few other things.. output of pages is cached, any page can depend on data from multiple other pages, output of a page can be reused multiple times and filewatchers ensure a script is reevaluated on change and all steps that were invalidated run again combine that with intellij idea as the IDE and tornadofx visualization and some static file outputs.. i think it would be something i'd be happy to use
a
Indeed, but this project is impossible to manage with one or two people team. I think it is important to separate back and front-end. If you have good back, than you can use it for jupyter and then create your own front. My team has some plans for such front, but it is a huge task.
Kotlin is currently not very appealing for scientific people due to lack of proper visualization. We need to create something that will just work as soon as possible.
n
data2viz is getting somewhere i think
but yes the visualitatiuon is probably where this will make a difference or drown
a
Not sure about it. The charts are propreitary which is not acceptable for science and you need some kind of environment to run them. Just remember that scientists are not programmers.
Beakerx is a good platform, so any help with it would be appreciated.
n
well i sarted this project as a test for how much kotlin scripting is ready to use .. and one of the features there is the excellent IDE integration can i work with beakerX having the notebook open in idea ?
a
I don't know. Never tried it. It is just regular jupyter notebook.
n
well either way feel free to pick my brains about scripting, but i have not really plans for making a jupyter thing in the near future but i am sure the core code can be ported to BeakerX
t
If you're playing with notebooks for the first time, this is worth watching

https://youtu.be/7jiPeIFXb6U▾

n
yeah i saw that one.. it partially inspired me that i need to make something i would actually want to use
a
Thanks @thomasnield! We're making some good progress. We'll be sure to show you when we're a little further along
a
By the way, I also do not like notebooks. The problem is that IDEA is too heavy weight to do quick-fix things that are usually needed. Also, a notebook has this markdown rendering capabilities that allows to document the process as it goes. What I would really love is a notebook with fixed cell dependencies, where changing something above, automatically invalidates everything below. I think it is possible to achieve that behavior with dataforge-like executor and plain jupyter notebook, but I just don't have time for it now.
☝️ 1
n
change something and it invalidates and reruns all cells (i call them pages) below it? seems familar.. i implemented that yesterday
now the next big hurdle is to get the UI going
o
It's a humble opinion, but couldn't a standalone Kotlin specific editor plus C/C++ support could be more useful ?
a
@otakusenpai Not sure what are you talking about and what does C++ have to do with this
o
idk, I'm quite new to Data Science. Though C++ doesn't have a need here
How do you apply faster computation in higher level languages such as kotlin? I have had 4 years of C++ experience but it was mainly normal stuff(just C++11, not much STL besides strings and vectors and a few other algos). Basically what I learnt on my own.
There is JNI, which can help in accessing native computing power
a
You should understand that "fast programs in C++" is for most part a myth. One can write something super-fast in C++, but not your general level programmer. In my experience, scientific programs in C++ in most cases are slower in real life just because people who write them are not that experienced. Latest JVM test show that without specific code optimization it is as fast or even faster that native program in C++. The exceptions are heavy duty super-optimized native libraries like tensor flow, but you can use JNI for them.
You can count the number of libraries you need native-base acceleration for by fingers on one hand... I can remember only tensorflow, because matrix algebra in libraries like ojalgo or EJMA are already as fast as native BLAS.
o
Hmm.... I see that your kmath library is fully Kotlin based.... yeah i can agree that using Kotlin is more type safe and generally productive to use than C++....
a
It is not in fact fully kotlin based, I t just does not yet have native part which will depend on native libraries. It has an interconnection with koma, which does have some native wrappers.
o
Hmm
a
Koma also has some wrappers for native BLAS libraries, but recent study shows that on JVM one just does not need them with JDK 11.
o
Does the probability part depend on Statistics and Probablity? Like Poisson or Bernoulli(stuff taught to us during 2nd year). I'm not that great in Maths though.
a
The probability part of kmath is not implemented yet. But yes, it will include distributions and a lot more since mathematical statistics and Bayesian methods are my specialty.
I have them in older project and will gradually migrate everything into multi-platform kmath.
o
Yeah I too want to install JDK 11. But I'll be using some other JVM like Eclipse OpenJ9 or something like that.
Hmm noice 😬
a
There is OpenJ9 for JDK 11 in adoptopenjdk package. But I did not have time to play with it yet.
I don't know if it supports nice vectorization features from oracle pack
n
looks cpp does not actuallyc unlock any faster processing
its not gonna make your program magically speed up and its not suited for scientists
a
@Nikky it unlocks better flexibility. You can do things faster in c++, but in most cases there are a lot of ways to things slower and bugier.
n
writing code is definitly slower
a
correction: You can do things faster in c++ if you now how and have a lot of time to debug everything
n
and that why python and R are so poular with data scientists i guess
a
yep
n
but writing code in kotlin is fast too,.. the problem is all the setup around making a kotlin project
gradle and whatever
well cpp is very much offtopic.. this thread cwas supposed to be aboutsome sample to showcase our notebook with
a
yep. Slack really needs functionality to split threads
2 Views
Open in Slack
PreviousNext
Powered by