We are experimenting with notebooks. Is there a wa...

# datascience

[Paul Woitaschek]

We are experimenting with notebooks. Is there a way to share code across multiple ipnyb notebooks?

[altavir]

You can load any Jas file or a directory with classfiles into notebook via @file:DependsOn directive

[Paul Woitaschek]

That doesn’t seem to work

[Paul Woitaschek]

Okay that was a different issue, it’s still not resolving:

[altavir]

You can't load kts. You can load compiled jar or classes

[altavir]

Here are full docs on library loading: https://github.com/Kotlin/kotlin-jupyter/blob/master/docs/libraries.md

[altavir]

By the way @Ilya Muradyan, great work!

[Paul Woitaschek]

Thanks! That’s a long document. It would be great if there were samples for this.

Our current use case is:
We are migrating away from apache zeppelin. My plan is to create a repository for all analysis that we use for dev internal stuff.
In the end we will use a jdbc connection and then transform the results from a sql query to a dataframe.

In the solution I’m picturing we’d have a src/main/kotlin structure where common queries and parsing is located.

And then we’d have a bunch of single-analysis targeted ipynb’s which make use of these files.

Does that concept make sense with the way these ipynb’s and generally kotlin notebooks are set up?

[altavir]

I had to spend few minutes searching, but I found proper documentation for annotation-based imports: https://github.com/Kotlin/KEEP/blob/master/proposals/scripting-support.md#kotlin-main-kts

[altavir]

You actually can use external kts via @file:Import directive. You can also write a direct path to a jar inside DependsOn , but it must be compiled jar.

[Paul Woitaschek]

Thanks! I’m still missing the greater picture here. How is this designed to work with some form of a locally built, reusable code infrastructure?

[altavir]

Let me find a demo...

[altavir]

Here it is: https://datalore.jetbrains.com/notebook/ptQDfQAcrjNxzIO0AEqovZ/rDxhvLe6OXRYcwgp3vz4OO. I use a jar file compiled from a project. I also can use an artifact, deployed to mavenLocal (it is actually a more reliable way, because then I don't have to think about dependencies).

[Paul Woitaschek]

What I found is all self contained:

https://github.com/Kotlin/dataframe/tree/b2f96b2e38c396230ce19e82f1368ee9f94813bf/examples/notebooks
https://github.com/Kotlin/kotlin-jupyter/tree/master/samples

[Paul Woitaschek]

Should that link from above be accessible without any kind of login?

[altavir]

The best way is to:
• Publish artefacts from your project to mavenLocal
• Use @file:Repository("*mavenlocal")
• Use your reusable code as a regular dependency.

[altavir]

It requires free datalore account. But I recommend it anyway if you want to play with kotlin notebooks.

[altavir]

This is example for this talk:

https://youtu.be/4Sg2Qju67kE▾

(the talk itself is in Russian, but I guess, you can read subtitles.

[Paul Woitaschek]

Do you think there is any way to automate that part? I’d like to use it an edit, then press run replacement.
I’d like to find a way to not write documentations for my devs like:

If you edit the files there, you need to run this script and then later update your notebooks

[Paul Woitaschek]

Maybe some process.execute(./gradlew build) and then a file:import as the header of each noteobooks?

In an ideal world I’d just have a structure like

./
├── build.gradle.kts
└── src
    └── main
        └── kotlin
            ├── helper.kt
            └── notebook1.ipynb

[altavir]

It depends on what is the process for your common code. If is some kind of library with regular development cycle, it is already automated. You deploy new version to the repository and then ask your scripters to update version number and reload notebooks. They do not have to do anything else.

[altavir]

and yes, you can run gradle build (probably better to use gradle publishToMavenLocal) from notebook.

[altavir]

But I have a feeling is what you need is actually idea-based notebooks: https://plugins.jetbrains.com/plugin/16340-kotlin-notebook. They work in project classpath already. You do no need to reload anything. It is still in development, but you can try it.

[altavir]

@roman.belov can probably tell more about it.

[Paul Woitaschek]

Yes! That sounds way more like what I want

[Paul Woitaschek]

Now we’re talking!

[altavir]

On the plus side, no Python at all. But I still can't make it work properly on MPP projects. It is in active development though.

[Paul Woitaschek]

I did some magic for our upcoming department days and managed to convert a git commit history to a dataframe. This is really great stuff!

[Paul Woitaschek]

This is so nice!