https://kotlinlang.org logo
Join Slack
Powered by
Hey everyone :wave: I’m working on a Dokka Plugin...
# dokka
s
Hey everyone 👋 I’m working on a Dokka Plugin, and I’m trying to gather all code-fences. It seems that when I drill down in 
modules: List<DModule>
from 
PreMergeDocumentableTransformer
(trying to do it as soon as possible in Dokka when info is available) I end up with 
DocTag.CodeBlock
but that is again nested with more 
CodeBlock
. Trying to reconstruct the original code fence from the 
CodeBlock
feels a bit off. Is there a way I can collect the 
CodeBlock
as a simple 
String
?
m
I am not entirely sure what is the issue there. Expected output should more or less look like this: https://github.com/Kotlin/dokka/blob/355acd296ac16902f588a879e99efd548f0dc0e7/plugins/base/src/test/kotlin/markdown/ParserTest.kt#L951 Could you give a sample of kdoc that you are passing to dokka?
s
It looks like this: https://github.com/Kotlin/dokka/blob/355acd296ac16902f588a879e99efd548f0dc0e7/plugins/base/src/test/kotlin/markdown/ParserTest.kt#L970 But if I want to compile that snippet, or run it with ScriptEngine than I need a String not a list of CodeBlock. So I would need to translate the CodeBlock list of children back to a String. I’m wondering if I can access that raw String somewhere instead, because transforming all CodeBlocks back to String seems like a huge waste. I’m trying to replace a tool we have now for compile checking code inside KDoc that runs before Dokka into a Dokka plugin.
I’m using the Dokka Plugin Template, and added a simple KDoc on top of the classes under test in the test folder.
m
No, there is no way to get the string from dokka without hacks. The reason being is that we process Kdoc and Javadoc so we need some common abstraction. In your case i’d just write converter if you really need it
s
That’s a shame that I cannot access the raw string anywhere. Do you know if it’s possible to have anything other than 
P
and 
BR
in 
CodeBlock
?
m
From our side i’d say no, but there is nothing preventing other plugin creators from adding their other / own nodes there.
👍 1
s
Thank you for the feedback
I will share in the channel if I have a working prototype. I’m hoping to replace this tool: https://github.com/arrow-kt/arrow/tree/main/arrow-libs/ank
m
Oh, nice I can tell you how i’d do some parts of this project regarding code checking. Code in documentation is available in 2 places: in doc tags model (thats those code blocks that you were talking about) and samples are added by 
SamplesTransformer
. If you were to check code that is passed in those 2 places i’d create a transformer for documentables and another one for pages. This transformer would look at the code and compile it using our analysis (this is an extension point in 
DokkaBase
, you can check out the 
SamplesTransformer
on some example on how to use it). After you have your analysis working, you need to get an error stream (somehow).
I’ve created an issue regarding the code blocks, since i agree that we don’t need any other tags inside them (after all they represent preformatted block of code): https://github.com/Kotlin/dokka/issues/2042
s
Thanks so much for the additional feedback! Awesome, thanks for the ticket. I updated to Kotlin 1.5, and I think the template is broken. Shall I create a ticket somewhere for it?
I tried bumping to 
Kotlin 1.5.20
and 
Dokka 1.5
but it’s now results in 
java.lang.NoSuchMethodError
within Dokka.
Untitled.txt
Untitled.txt
m
Woops, please do i’ll fix it
🙏 1
s
Oh, I cannot create issues on a template repo so I created it on Dokka. https://github.com/Kotlin/dokka/issues/2043
I ran into another issues now 😅 but making a lot of progress! Currently I am trying to get 
ScriptEngine
running.
There seems to be an issue with including these dependencies in the same project as a Dokka plugin.
Copy code
implementation("org.jetbrains.kotlin:kotlin-compiler:1.5.21")
implementation("org.jetbrains.kotlin:kotlin-script-util:1.5.21")
runtimeOnly("org.jetbrains.kotlin:kotlin-reflect:1.5.21")
runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-compiler:1.5.21")
I think 
"org.jetbrains.kotlin:kotlin-compiler:1.5.21"
is the problematic one, but I’m not 100% sure. It’s the only dependency shared with Dokka, 
kotlin-reflect
is only used in the Gradle plugin. Tried to exclude it from 
"org.jetbrains.dokka:dokka-base:$dokkaVersion"
but that also didn’t work. I’m getting following 
NoSuchMethod
error without reaching my extension point.
Untitled.txt
Untitled.txt
If I don’t include it myself, then it fails with the following when trying to evaluate 
val x: Int = 1
in the 
ScriptEngine
.
Untitled.txt
Untitled.txt
I hope to make some time soon to try and debug this to figure out what’s going on. If you have anymore insights it’d be really great.
I seem to keep running into dependency/runtime issues when trying to get 
ScriptEngine
to run within Dokka. Seems to be because Dokka is hooking into the compiler phases, by injecting its own service? But I’m not 100% sure, I don’t have such deep knowledge especially after all the FIR/IR changes. Can anyone guide me in the correct direction? I think I’ve tried a mix of about all dependencies/exclusions that I could think of. There is more info in the thread.
Copy code
'org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.createForProduction(com.intellij.openapi.Disposable, org.jetbrains.kotlin.config.CompilerConfiguration, org.jetbrains.kotlin.cli.jvm.compiler.EnvironmentConfigFiles)'
java.lang.NoSuchMethodError: 'org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.createForProduction(com.intellij.openapi.Disposable, org.jetbrains.kotlin.config.CompilerConfiguration, org.jetbrains.kotlin.cli.jvm.compiler.EnvironmentConfigFiles)'
m
Eh, why do you need to add second compiler to dokka (we use CLI one)? We already have one. Could you look at 
dokka-analysis
module and see if your use-case is not covered by that?
s
I’d like to evaluate snippets within the plugin, such that you can write tests as docs. I.e.
Copy code
/**
  * Maps every elements of the [List] with [transform]
  *
  * ```kotlin:ank
  * import io.kotest.assertions.shouldBe
  *
  * listOf(1, 2, 3)
  *   .map { it + 1 } shouldBe listOf(2, 3, 4)
  *
*/ fun <A> List<A>.map(transform: (A) -> B): List<B> = ...```
There is serveral different options possible. • kotlin:ank -> compiles and evaluates • kotlinankplayground -> compiles, evaluates and inserts HTML code for support Kotlin Playgrounds • kotlinankcompile -> Only compiles, doesn’t evaluate Any I hope to support a lot more things in the future, but this requires an instance of 
ScriptEngine
. (This currently support Java & Kotlin in the “legacy” tool)
Afaik there is nothing in analysis or in the compiler that can help me achieve this, so my only option currently is to rely on 
ScriptEngine
.
m
Okey, tbh i dont know what happens here I’ve asked a buddy from JB, we will see
s
Thank you @Marcin Aman!
i
@simon.vergauwen I do not understand the whole situation yet, but it seems that you're adding dependencies to the internals of the compiler to the dokka classpath, that already contain some of them, likely in the relocated form. And also the way that you're using to get the Kotlin JSR-223 scripting engine is obsolete and deprecated. So first I suggest to try to use dedicated JSR-223 implementation from 
kotlin-scripting-jsr223
jar instead of using these deps:
Copy code
implementation("org.jetbrains.kotlin:kotlin-compiler:1.5.21")
     implementation("org.jetbrains.kotlin:kotlin-script-util:1.5.21")
     runtimeOnly("org.jetbrains.kotlin:kotlin-reflect:1.5.21")
     runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-compiler:1.5.21")
and registering the engine via 
src/main/resources/META-INF/services/javax.script.ScriptEngineFactory
. You only need to add
Copy code
runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jsr223:1.5.21")
instead, and hopefully it will work right away. If not - please ping me again, I'll need to dig a bit deeper then.
s
Oh, I didn’t know there was a new dedicated jar for that. Thank you so much for the fast reply! 🙏 I will keep you posted @ilya.chernikov
@ilya.chernikov Okay, so I tried adding 
runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jsr223:1.5.21")
but simply having that makes Dokka fail. Also on the default Dokka Plugin Template, it’s reproducible by simply adding 
runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jsr223:1.5.21")
.
Untitled
Untitled
i
Ok, I see. I still haven't looked into details - I need to understand now how dokka imports compiler internals, but from the stacktrace I can guess that it strips some parts that are important for the scripting engine, therefore it fails. Can you try to check the classpath at the point there you're requesting the engine?
m
s
I can’t even reach that point. Just having that on the classpath make Dokka crash before it reaches any extension points. I’ll try to set a breakpoint before it crashes, and check the classpath.
i
Hm, interesting. @Marcin Aman, do I see it correctly that you're not relocating any of the compiler packages?
m
No, i dont think so
i
@simon.vergauwen could you try to use 
kotlin-scripting-jsr223-unshaded
instead of the regular one?
s
This is the classpath at runtime when Dokka is running right before it crashes.
i
And you probably need to exclude dependency to the 
kotlin-compiler
from transitive dependencies of the jsr-223 jar.
But looking at it, it seems to me that it could be quite tricky. The way how compiler is imported into the dokka will make it quite difficult for plugins that will attempt to import e.g. scripting. Or anything based on the IntelliJ SDK, for that matter. Gradle had similar issues then they started to embed kotlin compiler into their code, and they ended up in some very sophisticated isolating classloaders for plugins. Maybe it is a good idea to implement some shadowing of the kotlin and intellij stuff in dokka.
s
When I include 
kotlin-scripting-jsr223-unshaded
it doesn’t crash while starting Dokka anymore, but the dance with errors have begun again when touching the script engine.
This happens when I use:
Copy code
runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jsr223-unshaded:1.5.21") {
    exclude("org.jetbrains.kotlin", "kotlin-compiler")
}

// wired to Dokka
object AnkCompiler : PreMergeDocumentableTransformer {
    val classLoader = URLClassLoader(emptyList<String>().map { URL(it) }.toTypedArray())
    val seManager = ScriptEngineManager(classLoader)
    val engine = requireNotNull(seManager.getEngineByExtension("kts")) { "getEngineByExtension" }
    override fun invoke(modules: List<DModule>): List<DModule> =
        modules.also { engine.eval("val x: Int = 1") }
}
& 
kotlin.script.experimental.jsr223.KotlinJsr223DefaultScriptEngineFactory
in META-INF.
Untitled
And I end up with this one again if I simply use 
runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jsr223-unshaded:1.5.21")
without excluding the compiler dependency. Same code. All this testing is done in the Dokka Plugin Template, to have a clean testground.
Untitled
i
Please, drop 
kotlin.script.experimental.jsr223.KotlinJsr223DefaultScriptEngineFactory
in the 
META-INF
, you don't need it at all with the new jsr-223 implementation.
Although it will not necessarily help.
s
Nope, didn’t help. Thanks for all the help already
Seems like it’s going to be tricky
i
I will have a closer look in the next few days, maybe I'll be able to find a trick. In general, the dokka includes 
kotlin-compiler
jar and shadows it as 
kotlin-analysis-compiler
(as Marcin mentioned above - https://github.com/Kotlin/dokka/blob/master/kotlin-analysis/compiler-dependency/build.gradle.kts and as visible in the classpath too). In this form the only easy possibility is to convince jsr-223 jar to use it as a dependency instead of the regular 
kotlin-compiler
. And it may work only with 
-unshaded
jsr-223. Theoretically I cannot yet imagine why it shouldn't work, so I'd try it. Otherwise some isolation is needed.
s
Okay, I see. That explains what is happening, thank you for explaining. If you have any findings I’d love to know, I’m excited to get this project started and get rid of all the legacy documentation tools we have atm.
i
Hi @simon.vergauwen Sorry that it took me so long, but I managed to make the simple example run with your plugin attached just by using the following way to depend on the jsr223 jar:
Copy code
runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jsr223-unshaded:1.5.21") { isTransitive = false }
(turning off transitivity doesn't work for me with the 
kotlin
helper, I haven't checked how to deal with it). Please try it in more advanced scenarios.
s
Hey @ilya.chernikov, Thank you so much! That’s awesome. I’m going to try and finish the POC for my use-case ASAP and I’ll keep you posted!
@ilya.chernikov were you able to evaluate any code inside the 
ScriptEngine
? I updated my code, and when I try to evaluate a simple snippet. 
val x: Int = 1
or 
println("HELLO")
it crashes. Attempted both with -and without 
kotlin-std-jdk8
loaded into the classpath of the 
ScriptEngine
.
👀 1
i
@simon.vergauwen, I hope I nailed it from the second attempt, sorry about the first misleading one. Maybe my solution is overcomplicated, therefore I'll explain problems that I found one by one and solutions I applied, so maybe you'll be able to simplify something: • first - you need to use kotlin jsr223 1.5.0 implementation, as this seems the one used in dokka 1.5.0, and, as I said before, as we're trying to reuse the compiler from dokka classpath, we need exact version match here. The problem you posted above about missing 
PackageViewDescriptorFactory
class arises due to the mismatch.
• second - the classpath that you need to add to the plugin to be able to load script engine and reuse dokka-bundled compiler is of course nowhere as simple as it seemed to me in the first attempt. I ended up with (definitely too precise) this one:
Copy code
runtimeOnly(kotlin("reflect"))
    runtimeOnly(kotlin("script-runtime"))
    runtimeOnly("org.jetbrains.kotlin:kotlin-script-runtime:1.5.0") { isTransitive = false }
    runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jsr223-unshaded:1.5.0") { isTransitive = false }
    runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-common:1.5.0") { isTransitive = false }
    runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jvm:1.5.0") { isTransitive = false }
    runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-jvm-host-unshaded:1.5.0") { isTransitive = false }
    runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-compiler:1.5.0") { isTransitive = false }
    runtimeOnly("org.jetbrains.kotlin:kotlin-scripting-compiler-impl:1.5.0") { isTransitive = false }
this is quite likely could be simplified, either by depending on the 
jsr223
one and excluding only 
kotlin-compiler
(I keep forgetting gradle voodoo around dependencies handling), or by using some dependencies that do not depend on the compiler transitively.
• third - with this I still got classloading problems until I created the classloader for the engine explicitly.
Copy code
val classLoader = AnkCompiler::class.java.classLoader.let { it as? URLClassLoader }?.let {
                URLClassLoader(
                    it.urLs.filter {
                        it.file.contains("/kotlin-script") ||
                                it.file.contains("/kotlin-stdlib") ||
                                it.file.contains("/kotlin-reflect") ||
                                it.file.contains("/kotlinx-coroutines") ||
                                it.file.contains("/kotlin-analysis-compiler")
                    }.toTypedArray(),
                    null
                )
            }
            System.setProperty("kotlin.jsr223.experimental.resolve.dependencies.from.context.classloader", "true")
            Thread.currentThread().contextClassLoader = classLoader
            ScriptEngineManager(classLoader).getEngineByExtension("kts").eval("println(31313131)")
I wasn't able to understand exacly what is going wrong here, maybe there is ann easier way to achieve the same isolation level. Note the line that set obscure system property - this is optional and experimental, but may save some memory. And setting the context classloader is needed for the engine to correctly define the compilation classpath.
• and on top of that - since it seems necessary to create isolated classloader anyway, it might be possible to remove dependencies to the jsr223 and related jars from your plugin and specify it explicitly in the classloader. But then you probably need to use some more gradle voodoo to be able to download the dependencies by gradle, but not include them into runtime classpath , but pass somehow to your plugin explicitly.
s
Awesome, thank you @ilya.chernikov for the very detailed explanation and hints & tricks🙏
@ilya.chernikov thank you again! I got it working https://github.com/nomisRev/AnkDokkaPlugin/pull/1/files#diff-9fd4d89c05f288aafdbf91e328790aa9d7f7e5e34709252184ee716bee44a88dR31 Soon first release of compiler-checker for code snippets inside KDoc through Dokka 🥳 With an evaluation step, so you can write tests inside your documentation
👍 1
@ilya.chernikov it works if you manually load js223 onto the classpath before calling the ScriptEngine, but the resulting solution to manually load the jars for JS223 is more cumbersome. So I’m going to stick to your original solution! Here is the branch where I ran with JS223 jars. https://github.com/nomisRev/AnkDokkaPlugin/commit/f7c7af1c59076874b9324582878c163633dcfb9a
i
@simon.vergauwen to be honest, I like this solution with dedicated configuration more, it looks much less fragile than my original one, because it doesn't list particular jars and file name parts all over the places. But it's up to you, of course.
s
Thanks for the feedback @ilya.chernikov. It’s the Gradle voodoo I’m not happy with.. Besides that this is definitely better for the reason you mention It currently works with a manual task to re-download the dependencies, which in turn relies on a custom Gradle config and a shell script.. And it’s all being checked into source control. 😅
So guess I need to work on my Gradle a bit more ><
5 Views
Open in Slack
PreviousNext
Powered by