kotlinlang #datascience

Hi there, What about missing values in kotlin? I constantly struggle with this question when working on krangl. For double we could follow the pythonic way and treat NaN as missing value, but for integers there is no equivalent. For sure, nullability is a core feature of the language, but where possible it feels wrong to me (from a performance and memory perspective) to replace IntArray with Array<Int?> to support missing values as null (which would be most pretty though). Any idea, KIPs, etc? To give another example,

Double.NaN.toInt()

evaluating to 0 is just plain wrong (and a bug imho). (

NaN.roundToInt

at least throws an exception)

Slackbot

06/26/2020, 8:16 PM

This message was deleted.

holgerbrandl

07/02/2020, 10:21 PM

Some more baby steps toward data science with kotlin https://github.com/holgerbrandl/krangl/blob/master/CHANGES.md#v013 🙂

👏 7

Vladyslav Sitalo

07/04/2020, 1:45 AM

In the Data Science podcast https://talkingkotlin.com/catching-up-with-data-science-at-jetbrains/ @roman.belov talks about “type providers” I found the concept fascinating but I wasn’t able to find any information about Kotlin implementation. Does anyone have pointers about that?

altavir

08/14/2020, 8:57 AM

message has been deleted

🎉 6

altavir

08/15/2020, 4:04 PM

I've just spent some time helping a student with Julia build fix. Guys, I am really happy, that we did not take it as a primary language, when we had a choce (took Kotlin instead). It is not that the lagnuage is bad (yet there are some really quesrionable solutions), it is tooling and infrostructure (just who thought that using github commits as versions is a good solution?). Julia is considered popular in data science and finance. I think kotlin is much more convenient.

👍 9

altavir

08/27/2020, 12:59 PM

Kotlin-spark announce: https://blog.jetbrains.com/kotlin/2020/08/introducing-kotlin-for-apache-spark-preview/

altavir

08/27/2020, 1:24 PM

@Maria Khalusova a few comments about the https://github.com/JetBrains/kotlin-spark-api/wiki/Quick-Start-Guide#building-the-application-with-gradle. • Why do you need shadow? Is it required for basic build? • As far as I can understand, kotlin stdlib dependency is not required anymore for kotlin 1.4+. • What about kotlin-jupyter?

bjonnh

09/02/2020, 11:53 PM

Ok the only thing I'm missing is that I want the "show source code/show definition" I have in IntelliJ and better autocompletion… 😄

Steven Wang

09/04/2020, 1:14 AM

I can not figure out how to create a pivot table from dataframe in krangl (like pd.DataFrame.pivot_table function), any hint will be appreciated!

Florian Magin

09/06/2020, 3:08 PM

Hey, I remember this Slack Channel being referenced somewhere related to the Kotlin Jupyter project. Has anyone done any working on embedding the Kotlin Kernel into another application context?

Florian Magin

09/07/2020, 11:33 AM

So I got the Jupyter Kotlin kernel running fairly quickly but I am now running in some issues that might be related to the kernel internals: I start

IkotlinKt.main

like the python code in jupyter would, but I call it from java code. This allows me to attach a jupyter-console to the kernel and access all the packages. But the classes inside the REPL do not share the static class variables with the java code that started the kotlin kernel via

IkotlinKt.main

. My guess is that this is accidental because this would never be relevant if you the kernel is supposed to be started by Python anyway. Does someone have a quick pointer of the relevant documentation or code concerning those internals?

Florian Magin

09/07/2020, 12:14 PM

I am also getting the error

ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider org.jetbrains.kotlin.mainKts.jsr223.KotlinJsr223MainKtsScriptEngineFactory could not be instantiated

in the kernel output which is fairly vague. Is there a way to get a more concrete error?

Steven Wang

09/07/2020, 2:45 PM

Krangl question:

val df = dataFrameOf("strcol") ("s1", "s2", "s3")
val l = listOf("s1", "s2")
// Is there any vectorized version to achieve this filter?
df.filterByRow { l.contains(it["strcol"]) }

Maria Khalusova

09/09/2020, 2:23 PM

Hi folks, if you plot your data, I'd love to learn about how you do it. Let me know if I can DM you with a couple of questions.

✋ 1

Florian Magin

09/11/2020, 2:55 PM

I am currently trying to understand how the

implicitReceivers

can be used. Are they supposed to be changeable at runtime?

Florian Magin

09/11/2020, 3:02 PM

implicitRecievers

of the the compiler is passed the list of classes, and

implicitReceivers

of the evaluator is passed the list of objects?

Florian Magin

09/11/2020, 3:04 PM

Also: Is there a name for this kind of pattern:

val x = ScriptEvaluationConfiguration {
    implicitReceivers(builder)
}

I think I don't fully understand what this is a short form for

Florian Magin

09/11/2020, 4:04 PM

Does the kernel have a magic variable to get the last cell result? Like Pythons

x = _

Florian Magin

09/11/2020, 4:05 PM

or even how Jupyter with Python uses

x = _2

to reference specific cells and not just the last

Steven Wang

09/16/2020, 12:32 PM

any good library support load/write data from/to disc available? e.g. pandas has support to write feather format (I use a lot) or hdf5 file (I use a lot too), I search around for few days, seems could not find any library support that ...

Basayya Kulkarni

09/25/2020, 1:11 PM

Hi what is best way to chunk the CSV file into multiple files and process the data for 1 or 2lack of records in Kotlin.as I know we can do it in Pythan using Pandas chunking data.

Basayya Kulkarni

09/25/2020, 1:23 PM

I was working converting R code to Kotlin and I Identified that in R datatable is very faster than the Dataframe,Could you please advice in Kotlin what is best performance for huge data which has more functionality like Pandas gives us.

Tilman Krokotsch

09/27/2020, 2:31 PM

I played around with DL4J a bit but found the syntax for declaring neural networks a bit lacking. Too Java for my taste. As I am currently trying to pick up Kotlin, I coded a small "library" to help with that using the type-safe builder patterns from the tutorials. You can find it on GitHub or read about it on my blog if you are interested.

:kotlin-flag: 4

🤘 1

Steven Wang

09/29/2020, 5:11 PM

krangl question: I could not find equivalent of pandas.Dataframe.shift(-1) function so I am using following code to do that. any better way or any function available should use?

df.addColumn("label") {it["range"].asDoubles().drop(1) + listOf(Double.NaN)}

Florian Magin

10/02/2020, 1:06 PM

One feature I am currently looking into is the

Shift+Tab

Shortcut in Jupyter that displays a tooltip (with the documentation) for the object at the current cursor. I think this doesn't work with Kotlin yet, but would be a great feature. Has anyone looked into this yet?

Florian Magin

10/02/2020, 1:11 PM

The core issue will most likely be to actually gather the information to display there, because this isn't contained in the JVM byte code unlike python docstrings and parameter names. One option could be to add an extra field to the library descriptions that specifies something (a URL to the doc? a class that handles it?) that is then used to resolve the documentation

Michal Harakal

10/21/2020, 7:33 PM

I am playing again a little bit with jupyter-kotlin (wanted to test tensorflow kotlin library) and stumbled upon the same bug, which was reported in kotlin scripting https://youtrack.jetbrains.com/issue/KT-38212 Should I create a new bug or just add comment to that existing one?

Florian Magin

11/04/2020, 1:26 PM

I expect this to be an unusual question, but maybe someone can help me nonetheless: I have IntelliJ attached to the process that contains the Kotlin Kernel. Where would I need to set a breakpoint so all the locally available variables inside the Kotlin Jupyter Notebook with the IntelliJ debugger view? I find the latter vastly superior for inspecting objects, and traversing their referencing objects etc, but the Jupyter Notebook is much more pleasant to actually write code in to test out more complicated expressions

altavir

11/05/2020, 6:41 AM

As for debugging. I do not imagine why anyone will use full debugger for the notebook, but some debugging would be nice, so here is a kotlinish-notbookish solution: • Introduce a wrapper interface

DebugContext

like this:

interface DebugContext{
  fun <T> trace(value: T): ReadWriteProperty<Any?, T>

  (suspend) fun breakPoint()
}

• The idea is that you can wrap any variable in the context in a delegate and dump all changes to a debug container. When the

breakPoint

is called, the execution will block/suspend and dump current debug state to console/output widget and wait for user input to unblock. This could be implemented on a library level based on the notebook API. The usage will look like this:

%%debug
var a by trace(4)
repeat(10){
  a++
  breakPoint()
}

If you like this idea, I can write an issue from it. I personally prefere not to do complex code in the notebook and move all heavy logic to the project.

Title

altavir

11/05/2020, 6:41 AM

As for debugging. I do not imagine why anyone will use full debugger for the notebook, but some debugging would be nice, so here is a kotlinish-notbookish solution: • Introduce a wrapper interface

DebugContext

like this:

interface DebugContext{
  fun <T> trace(value: T): ReadWriteProperty<Any?, T>

  (suspend) fun breakPoint()
}

• The idea is that you can wrap any variable in the context in a delegate and dump all changes to a debug container. When the

breakPoint

%%debug
var a by trace(4)
repeat(10){
  a++
  breakPoint()
}

If you like this idea, I can write an issue from it. I personally prefere not to do complex code in the notebook and move all heavy logic to the project.

Ilya Muradyan

11/05/2020, 12:12 PM

Have you seen an implementation of debugger in python notebooks?

https://blog.jupyter.org/a-visual-debugger-for-jupyter-914e61716559

altavir

11/05/2020, 12:22 PM

No, probably I stopped working a lot with Python before it arrived. It looks interesting, but it seems it will require a lot of effort and actual JVM debugger attached to the process. Does scripting engine support source mapping?

View count: 2