Hey, I remember this Slack Channel being reference...

# datascience

[Florian Magin]

Hey, I remember this Slack Channel being referenced somewhere related to the Kotlin Jupyter project. Has anyone done any working on embedding the Kotlin Kernel into another application context?

[altavir]

Do you mean non-notebook or non-jupyter?

[altavir]

If you are talking about something non-notebook, than you can use kotlin scripting for that (jupyter kernel is built on top of it). As for other ipython environments, the work is being done with them as well. Jupyter is just the most popular environment, so it goes first.

[Florian Magin]

I am not sure if we are using the same terminology. For me, Jupyter is the project overall and the general architecture of kernels that you attach to. Notebook is the Notebook frontend (that has some issues like requiring that the notebook process started the kernel).
The thing I want to do, which might just work that easily is to start a thread in an application that invokes the main method of kotlin-jupyter-kernel.jar which I should then be able to attach too. I already tested that I can simply start this jar with java -jar ... in the same way the Notebook does and I will then have a kernel running that I can attach to with jupyter console --existing

[altavir]

If you want a console REPL, you do not need Jupyter at all. You can use Kotlin-scripting directly (and it gives a lot more freedom). Jupyter kernel is created specifically for ipython transport protocol server.

[Florian Magin]

ipython transport protocol server is somewhat the goal because this means I can attach a CLI terminal or qtconsole to it and don't need to implement the GUI inside the application myself

[altavir]

Ok, this is the second part of my answer. I know that currently work being done to use kotlin kernel for other ipython servers as well. Like Zepelin or Idea own ipython plugin. I think this work will be more intensive once the jupyter kernel is stabilized.

[Florian Magin]

But if kotlin-scripting is significantly easier overall I can use it as a first prototype. Could you point me to an example for kotlin scripting that is able to read in run arbitrary kotlin code and evaluate inside the full Java context available?

[Florian Magin]

What do you mean by "ipython servers"?

[Florian Magin]

For context: I have dealt with a lot of IPython related code (but only actual Python code) and have built projects already that embed a python kernel into an application that provided a Python interpreter and the possibility of Python plugins

[altavir]

ipython is a protocol. In order to use it, you need both the server and the client. Kernel is connected to the server via ZMQ pipe. Jupyter framework has both.

[altavir]

Kotlin scripting examples could be found here: https://github.com/Kotlin/kotlin-script-examples. For REPL you can see jupyter plugin source code, it is not very hard to read.

[Florian Magin]

Ok I think this is where I am getting confused, IPython for me is the simple "ipython" program that IIRC doesn't involve ZMQ or kernels

[Florian Magin]

jupyter-console looks like IPython on the surface but does different things in the background AFAIK

[altavir]

But how does the data from your evaluation gets to your browser?

[Florian Magin]

In what setup? With jupyter and python or with my vision of the kotlin setup?

[Florian Magin]

Or with pure IPython?

[altavir]

I am trying to explain how does Jupyter works. You need a web-server in order to communicate with Jupyter/Zepelin/whatever frontend. Your REPL code does not run any server.

[Florian Magin]

I think I can simply prove you wrong there

[Florian Magin]

I can run

java -jar /home/fmagin/.local/lib/python3.8/site-packages/run_kotlin_kernel/jars/kotlin-jupyter-kernel-0.8.2.61.jar -classpath=/home/fmagin/.local/lib/python3.8/site-packages/run_kotlin_kernel/jars/jupyter-lib-0.8.2.61.jar:/home/fmagin/.local/lib/python3.8/site-packages/run_kotlin_kernel/jars/kotlin-script-runtime-1.4.20-dev-2342.jar:/home/fmagin/.local/lib/python3.8/site-packages/run_kotlin_kernel/jars/kotlin-reflect-1.4.20-dev-2342.jar:/home/fmagin/.local/lib/python3.8/site-packages/run_kotlin_kernel/jars/kotlin-stdlib-1.4.20-dev-2342.jar:/home/fmagin/.local/lib/python3.8/site-packages/run_kotlin_kernel/jars/kotlin-stdlib-common-1.4.20-dev-2342.jar:/home/fmagin/.local/lib/python3.8/site-packages/run_kotlin_kernel/jars/annotations-13.0.jar /home/fmagin/.local/share/jupyter/runtime/kernel-641b8904-e16a-483b-be0b-58b6c73b8412.json -home=/home/fmagin/.local/lib/python3.8/site-packages/run_kotlin_kernel

[Florian Magin]

On my terminal. This is the copied process arguments that normally python -m run_kotlin_kernel /home/fmagin/.local/share/jupyter/runtime/kernel-641b8904-e16a-483b-be0b-58b6c73b8412.json would start

[altavir]

I am not sure, what you a trying to prove. Kotlin kernel is a stand-alone process, but it does not communicate with browser. It communicates with jupyter server. Here is the appropriate page from documentation: https://github.com/Kotlin/kotlin-jupyter/blob/master/src/main/kotlin/org/jetbrains/kotlin/jupyter/repl.kt

[Florian Magin]

I can attach to that kernel with jupyter console --existing=/home/fmagin/.local/share/jupyter/runtime/kernel-641b8904-e16a-483b-be0b-58b6c73b8412.json

[altavir]

Sorry, wrong link: https://jupyter-client.readthedocs.io/en/stable/messaging.html

[altavir]

Exactly. Jupyter ups the server and communicates with kernel via ZMQ

[altavir]

To summarize. Yes, Kotlin Kernel runs repl in a stand-alone process, but it designed to communicate with jupyter via jupyter-specific protocol (I am not sure if other ipython servers share the same protocol).

[Florian Magin]

And the thing I want to do is that my application starts the kernel instead. Then jupyter-console can attach to this kernel.

[Florian Magin]

No notebook process needs to be running for all of this

[altavir]

Yes, it should be quite possible right now.

[Florian Magin]

Do you have an expectation of how well that kernel will then be able to provide access of other libraries of the JVM it is running inside? Does importing simply work or are there some pitfalls to expect?

[altavir]

As I already said, kotlin REPL is working in a separate process, so you will be able to access the same functionality as in notebook (including plots in browser with plotly.kt 😊).

[altavir]

Just started kotlin kernel in console mode, it is fully functional. The only problem is that kotlin kernel dumps a lot of debug info into the console and bloats the output. @Ilya Muradyan, I think it should be somehow fixed in future.

[altavir]

And here is the plot!

[Florian Magin]

But my goal is that it isn't a separate process though and can access the entire API that the application provides

[Florian Magin]

While also running inside the context of the application.

[altavir]

I am not sure what you mean. Do you want to run Kotlin REPL inside kotlin application?

[Florian Magin]

I understand that my use case is probably really weird

[Florian Magin]

https://github.com/eset/ipyida

[Florian Magin]

https://github.com/fmagin/binja-ipython

[Florian Magin]

https://github.com/james91b/ida_ipython

[Florian Magin]

are some examples of this approach for other tools used in reverse engineering

[Florian Magin]

The tool I want to integrate the REPL into is https://github.com/NationalSecurityAgency/ghidra

[altavir]

If it is Java/Kotlin project, then you do not need to use IPython at all, just call kotlin REPL directly. Few years ago I did it with Groovy an Kotlin supports several ways to do that as well. Just see the link, I've shared below the thread, You just need to copy some code form that file without including any IPython machinery.
Also, isn't Hydra and NSA seems a bit... brrr..? I am a bit of Marvel fan.

[Florian Magin]

Also, isn't Hydra and NSA seems a bit... brrr..? I am a bit of Marvel fan.

I don't understand what you are saying with that. Ghidra is, as far as I remember, a reference to Godzilla

[Florian Magin]

I think I'll test a really basic kotlin REPL first then, and see if this works as I imagine it, and if not I will have more concrete questions to ask. Thank you for your time and patience in helping me 🙂

[altavir]

Kotlin REPL is in experimental phase, but is is fully functional. And there is a lot of information in #scripting about that

[Michal Harakal]

Discussion in TalkingPodcast points out some interesting informations regarding jupyter-kotlin … https://talkingkotlin.com/catching-up-with-data-science-at-jetbrains/

[altavir]

@Michal Harakal It is a little bit old. There is a more recent exciting presentation by Roman in JBR channel, but it is in Russian. There is also my article: https://levelup.gitconnected.com/a-first-dive-into-kotlin-jupyter-eab43dca9472. Both Kotlin-Jupyter team and module developers made a lot of progress since then. Still, the question is not solved by Jupyter in this case.

[Michal Harakal]

@altavir could you pls point me on that russian talk? I didn’t find it … BTW your medium article is very useful.

[Ilya Muradyan]

There is also my talk for team in Russian about kernel internals. I can share it if you wish

[altavir]

@Ilya Muradyan Oh, I would like to hear this one as well. I know how it works, but it is always interesting to hear interesting things again.

[Michal Harakal]

I am also interested in