is there some kind of standard realife task that is used fow notebooks ? i am writing something very similar that can process data in multiple steps and visualite it now it would be useful to copy some sort of sample usecase from somewhere that can be done using jvm libraries

Not sure I understand the question

well i started making my own completely seperate from jupyter, @amanda.hinchman-dominguez is doing the tornadofx vizualization we are just not happy with the need to run a webserver to run kotlin snippets

Jupyter has its limitations, but everyone use it and I am not sure it is possible to create a standalone visualization framework with comparable capabilities. Beakerx already has some visualization capabilities for JVM languages and, what is much more important, autotranslation feature to move data between python, and, say, kotlin. It seems reasonable to invest in it instead of creating something completely new.

also a few other things.. output of pages is cached, any page can depend on data from multiple other pages, output of a page can be reused multiple times and filewatchers ensure a script is reevaluated on change and all steps that were invalidated run again combine that with intellij idea as the IDE and tornadofx visualization and some static file outputs.. i think it would be something i'd be happy to use

Indeed, but this project is impossible to manage with one or two people team. I think it is important to separate back and front-end. If you have good back, than you can use it for jupyter and then create your own front. My team has some plans for such front, but it is a huge task.

data2viz is getting somewhere i think

Not sure about it. The charts are propreitary which is not acceptable for science and you need some kind of environment to run them. Just remember that scientists are not programmers.

well i sarted this project as a test for how much kotlin scripting is ready to use .. and one of the features there is the excellent IDE integration can i work with beakerX having the notebook open in idea ?

I don't know. Never tried it. It is just regular jupyter notebook.

well either way feel free to pick my brains about scripting, but i have not really plans for making a jupyter thing in the near future but i am sure the core code can be ported to BeakerX

If you're playing with notebooks for the first time, this is worth watching

yeah i saw that one.. it partially inspired me that i need to make something i would actually want to use

Thanks @thomasnield! We're making some good progress. We'll be sure to show you when we're a little further along

By the way, I also do not like notebooks. The problem is that IDEA is too heavy weight to do quick-fix things that are usually needed. Also, a notebook has this markdown rendering capabilities that allows to document the process as it goes. What I would really love is a notebook with fixed cell dependencies, where changing something above, automatically invalidates everything below. I think it is possible to achieve that behavior with dataforge-like executor and plain jupyter notebook, but I just don't have time for it now.

change something and it invalidates and reruns all cells (i call them pages) below it? seems familar.. i implemented that yesterday

It's a humble opinion, but couldn't a standalone Kotlin specific editor plus C/C++ support could be more useful ?

@otakusenpai Not sure what are you talking about and what does C++ have to do with this

idk, I'm quite new to Data Science. Though C++ doesn't have a need here

You should avoid writing anything C++ unless you have a lot of experience with it or you are creating games. Kotlin is very good language, but the problem that it lacks sufficient scripting and visualization support favored by "data scientists" for "quick and dirty" style programming. Bare bones compiler won't help you in that.

How do you apply faster computation in higher level languages such as kotlin? I have had 4 years of C++ experience but it was mainly normal stuff(just C++11, not much STL besides strings and vectors and a few other algos). Basically what I learnt on my own.

You should understand that "fast programs in C++" is for most part a myth. One can write something super-fast in C++, but not your general level programmer. In my experience, scientific programs in C++ in most cases are slower in real life just because people who write them are not that experienced. Latest JVM test show that without specific code optimization it is as fast or even faster that native program in C++. The exceptions are heavy duty super-optimized native libraries like tensor flow, but you can use JNI for them.

Hmm.... I see that your kmath library is fully Kotlin based.... yeah i can agree that using Kotlin is more type safe and generally productive to use than C++....

It is not in fact fully kotlin based, I t just does not yet have native part which will depend on native libraries. It has an interconnection with koma, which does have some native wrappers.

Hmm

Koma also has some wrappers for native BLAS libraries, but recent study shows that on JVM one just does not need them with JDK 11.

Does the probability part depend on Statistics and Probablity? Like Poisson or Bernoulli(stuff taught to us during 2nd year). I'm not that great in Maths though.

I have to work with some C++ libraries and it is my sincere belief that with current level of tooling the language is unsuitable for general purpose scientific development. Maybe if they get modules and repositories...

Yeah I too want to install JDK 11. But I'll be using some other JVM like Eclipse OpenJ9 or something like that.

There is OpenJ9 for JDK 11 in adoptopenjdk package. But I did not have time to play with it yet.

looks cpp does not actuallyc unlock any faster processing

@Nikky it unlocks better flexibility. You can do things faster in c++, but in most cases there are a lot of ways to things slower and bugier.

writing code is definitly slower

correction: You can do things faster in c++ if you now how and have a lot of time to debug everything

and that why python and R are so poular with data scientists i guess

yep

but writing code in kotlin is fast too,.. the problem is all the setup around making a kotlin project

err.. have yo tried to setup a project in C++? Let's start another thread.

well cpp is very much offtopic.. this thread cwas supposed to be aboutsome sample to showcase our notebook with

yep. Slack really needs functionality to split threads