Hello, I want to use Dokka in an open source multi...

# dokka

[louiscad]

Hello, I want to use Dokka in an open source multi-module, multiplatform and android project (all at once), and my attempts at setting up Dokka have failed (outputs empty doc), plus I'm super confused and overwhelmed by the documentation of Dokka.

Would someone that is familiar with Dokka setup join me for a pair-programming session where we get that setup working sometime tomorrow? I'm in CEST, but I have high flexibility.

FYI, the project in question is #splitties.

[Javier]

You have to add it to the root build.gradle.kts

[Javier]

import org.jetbrains.dokka.gradle.DokkaMultiModuleTask
import org.jetbrains.dokka.gradle.DokkaTask
 
plugins {
    id("org.jetbrains.dokka")
}
 
tasks {
    withType<DokkaTask>() {
        dokkaSourceSets.configureEach {
            includes.from(listOf("MODULE.md"))
        }
    }
 
    withType<DokkaMultiModuleTask>().configureEach {
        val dokkaDir = buildDir.resolve("dokka")
        outputDirectory.set(dokkaDir)
    }
}

[Javier]

That is my config

[Javier]

And run ./gradlew dokkaHtmlMultiModule

[Javier]

Writing code snippets in Slack mobile is just impossible, sorry

[louiscad]

Is configuring the DokkaMultiModuleTask tasks required?
Also, how am I supposed to tell I only want KDoc for projects under a certain hierarchy, and not the other ones?

[Javier]

I am not sure about that, I use it for all projects. Maybe there is a exclude function for that 🤔

[Javier]

About configuring that task, I don't know if there is a default value where dokka docs will be generated, probably yes

[louiscad]

You publish the doc for the sample modules as well?

[Javier]

I think I am having that problem at this moment too, I will have to fix it

[Javier]

Well, in my case, samples modules are not being published

[louiscad]

bring?

[Javier]

Sorry, mobile autocorrector

[Javier]

The only difference I can see is samples modules haven't a MODULE.md file

[louiscad]

Do you mean no publication happens if there's no Module.md file? Also, do I need to put some content in it?

[Javier]

And the published artifacts have the dokka plugin too

[Javier]

image.png

[Javier]

I don't know exactly what is blocking the generation of sample docs, but they are not there. I am not sure if it is caused by not using Module.md file, if it is caused by published artifacts are using dokka for generating the jar, or that samples are not using multiplatform plugin and the rest of projects are multiplatform

[Javier]

I am going to remove the dokka plugin in a published project to check if it is generated

[Javier]

Well, I can confirm you that the problem is that

[Javier]

You have to add the dokka plugin to the project you want to generate docs

[Javier]

and in root build.gradle.kts too (at least is what I am doing)

[Javier]

@Marcin Aman maybe can help here

[Zach Klippenstein (he/him) [MOD]]

you can configure a list of subprojects to generate for. Here’s an example

[louiscad]

I ended up putting this in an allProjects block:

plugins.withId("org.gradle.maven-publish") {

    plugins.apply("org.jetbrains.dokka")
    tasks.withType<DokkaTask> {
        dokkaSourceSets.configureEach {
            includes.from("README.md")
        }
    }
}

Plus the following in the root Gradle build file:

plugins {
    //... other plugins
    kotlin("multiplatform") apply false // Classpath concerns for dokka, might be unneeded.
    // See <https://github.com/Kotlin/dokka/issues/1810>
    id("org.jetbrains.dokka")
}

[louiscad]

Unfortunate outcome though 😭:

> Task :modules:experimental:dokkaGfmPartial FAILED
=============================== Gradle Doctor Prescriptions ============================================
| This build spent 40% garbage collecting.                                                             |
| If this is the first build with this Daemon, it likely means that this build needs more heap space.  |
| Otherwise, if this is happening after several builds it could indicate a memory leak.                |
| For a quick fix, restart this Gradle daemon. ./gradlew --stop                                        |
========================================================================================================

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':modules:experimental:dokkaGfmPartial'.
> Java heap space

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

BUILD FAILED in 9m 39s

[louiscad]

Retried after killing the daemon, same 😭 :

=============================== Gradle Doctor Prescriptions ============================================
| This build spent 44% garbage collecting.                                                             |
| If this is the first build with this Daemon, it likely means that this build needs more heap space.  |
| Otherwise, if this is happening after several builds it could indicate a memory leak.                |
| For a quick fix, restart this Gradle daemon. ./gradlew --stop                                        |
========================================================================================================
| The following operations were slower to pull from the cache than to rerun:                           |
| :modules:collections:jsIrGenerateExternalsIntegrated                                                 |
| Consider disabling caching them.                                                                     |
| For more information see: <https://runningcode.github.io/gradle-doctor/slower-from-cache/>             |
========================================================================================================

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':modules:preferences:dokkaGfmPartial'.
> GC overhead limit exceeded

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

BUILD FAILED in 8m 27s

Really bad first experience 😞

I'm gonna try bumping Gradle heap from 2GB to 3GB, but I find it insane that it's so memory inefficient.

[louiscad]

Dokka is really messing with my emotions!!
Here's the output after 6+ minutes of hitting my laptop CPU on request of the dokkaGfmMultiModule task 😡 🤬

[louiscad]

Here's the very changes where I try to set Dokka up: https://github.com/LouisCAD/Splitties/compare/LouisCAD:d0646e5...LouisCAD:0a25f9e

If you know how to fix it and want to submit PR, please target it against the wip-setup-dokka branch.

[Javier]

I will take a look in a few hours

[Marcin Aman]

I’ll also try to help, give me a sec ☺️

[Javier]

image.png

[Javier]

Doing allproject { apply(plugin = "org.jetbrains.dokka") } is not working for me

[Javier]

But adding to each project id("org.jetbrains.dokka") is generating the docs

[Javier]

I tried only with activities, but probably will work with all

[Javier]

image.png

[Javier]

Yeah, is it valid for you?

[Marcin Aman]

Wait, is there a reason why you dont want to use a build-in multimodule task like dokkaHtmlMultimodule or dokkaGfmMultiModule?

[Javier]

I am using dokkaHtmlMultimodule task for generating them

[Javier]

Do you want the setup with allproject or this is valid @louiscad?

[Javier]

Personally, I don't like the allproject config, I just prefer to create a precompiled plugin which can share the common config in all modules, for example, I have a precompiled plugin for publishing and that plugin has the dokka plugin too because I need it to generate de javadocs (MavenCentral requires them).

[Javier]

So I get something like this:

[Marcin Aman]

Oh because you applied it only on those 2 submodules. Thats weird, i didnt change it and am getting docs for all modules in the root build directory.

So i’ve also tried out this project and:

if (file("README.md").exists()) tasks.withType<DokkaTaskPartial> {
            dokkaSourceSets.configureEach {
                includes.from("README.md")
            }
        }

This won’t work since readme-s are not formatted correctly. They should be in a format defined here: https://kotlinlang.org/docs/kotlin-doc.html#module-and-package-documentation. Also in multimodule you are configuring DokkaTaskPartial not DokkaTask

As for the memory: i use 4G, it is quite a lot but the project is not a small one and it makes the build go faster (specially with some parallel workers).

Personally i’d add the split inheritors switch to configuration:

tasks.withType<DokkaTaskPartial> {
            pluginsMapConfiguration.set(mapOf("org.jetbrains.dokka.base.DokkaBase" to """{ "separateInheritedMembers": true }"""))
        }

Or by adding a base plugin to classpath:
https://kotlin.github.io/dokka/1.4.30/user_guide/gradle/usage/#applying-plugins
This will make an extra tab with members that are inherited so it is possible to see only those that are defined in certain class.
Personally i recommend using the html, since it works a little bit better for more complex projects (you get search + better navigation) and you can even use it when publishing to maven central

[louiscad]

I don't want to edit the build.gradle.kts files of every module for two reasons:
1. I'm lazy, and it's taking on my free-time
2. Generally speaking, all of the published modules should have KDoc, so I think Dokka should make that very common use-case super quick and easy to setup. Otherwise, it means the bigger the project, the most cumbersome it is to set it up. I know one can use Replace In Path, but that's still boilerplate, that make reviewing code more time consuming, with like, zero added value.

[Javier]

@Marcin Aman yeah, I am now generating for all

[Marcin Aman]

Also on the topic on memory:
we are aware about dokka being memory hungry and will try to improve it in the future. For now i recommend running with 4G and no deamon. Maybe also with some workers to speed things up: ./gradlew dokkaHtmlMultiModule --no-daemon --parallel --max-workers=3

[louiscad]

@Marcin Aman Why is the tool not telling me:
1. Not enough heap allocated, let's not try to fail at all, 3GB minimum are required (I'm not question memory inefficiency here)
2. The supplied files are improperly formatted, here is the error.

[Javier]

@louiscad If you are sharing a precompiled plugin in all projects, you haven't to edit every build.gradle.kts, only the precompiled plugin

[Marcin Aman]

@louiscad dokka will tell you if your files are not properly formatted but you configured wrong dokka task so the proper one didn’t know about those additional readme files. Here is an error message when i’ve changed DokkaTask to DokkaTaskPartial in your configuration:

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':modules:activities:dokkaHtmlPartial'.
> Unexpected classifier: "#", expected either "Module" or "Package". 
  For more information consult the specification: <https://kotlinlang.org/docs/reference/kotlin-doc.html#module-and-package-documentation>

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

[louiscad]

@Marcin Aman I'm suprised these memory concerns and workarounds are not stated in the Dokka documentation website at all.

I'd like to collaborate on the improvement of Dokka documentation because it was very overwhelming, and there's ambiguity with the website, the README and the GitHub Wiki. Should we start another thread, or discuss that privately, or a GitHub issue that'd be prioritized? I really don't wish anyone to have the same onboarding experience as me, that's why I'd like that documentation topic to get high priority.

[Javier]

@Marcin Aman I don't know if it is a issue or if it is the project config, but what I am seeing is that Dokka is generating docs for each module sequentially, instead of using parallelism

[Javier]

Is it going to be improved?

[Javier]

image.png

[louiscad]

@Marcin Aman How did I configure wrong Dokka tasks? Is there documentation that tells about right and wrong tasks to configure?

[Javier]

Well, org.gradle.parallel=false, but I don't know if Dokka supports parallelism btw

[louiscad]

IIRC, I disabled parallel Gradle because it wasn't working with Kotlin/Native compiler daemon. I believe this has been resolved in 1.5, but I cannot use 1.5 yet as a library with public API that committed to least breakages.

[louiscad]

I should have written a comment for that.

[Marcin Aman]

@Javier Dokka does multimodule by producing a template of the documentation for each module (in seperate child task), collecting outputs and filling blanks. This way handling of parallel operations is delegated to gradle. Ofc, under the hood we are also using coroutines to make it more efficient. In your example you only have 1 gradle worker so processing is sequential

[Javier]

Nice, then when @louiscad can enable it with 1.5 the build will improve a lot

[Javier]

image.png

[Javier]

I am not sure if it is stucked there 🤔

[louiscad]

I've seen Dokka apparently relies on compiling everything, a bummer for me as I build from CI, but would hardly be able to do so for docs building. I'm set back to pre-CI times 😔

[Javier]

What you mean? I use Dokka with GitHub Actions a lot, no problems for me

[Marcin Aman]

@louiscad If you didn’t find something in the docs feel free to contribute. We try to make them as good as possible but some things are obvious to us (people that work with dokka on daily basis) but are not for the new people. So every improvement that will make other people life easier is welcome 😄

[Marcin Aman]

Dokka doesn’t rely on whole compilation process but has to setup kotlin analysis so we can get information about code. This phase requires some artifacts fetching and parsing

[louiscad]

@Marcin Aman I do not want to send a PR to add what I miss, I want to talk about the doc structure, to make it not overwhelming. Do you have time to discuss that topic? Audio or video chat is also fine for me.

[Javier]

I canceled the task and I enabled parallel

[Javier]

image.png

[Javier]

image.png

[louiscad]

@Javier To reach that result, you had to edit all build.gradle.kts files, right?

[Javier]

You can run in CI something like ./gradlew dokkaHtmlMultiModule -P"org.gradle.parallel"="true"

[Javier]

Yeah

[Javier]

I can setup a precompiled plugin if you want sharing the common config

[louiscad]

@Javier The problem with running in CI is that unless I setup CI committing code, I'll not get history of the docs for future old versions of the project.

[Javier]

I am publishing docs for each version

[Javier]

for example

project.domain.com/
project.domain.com/spapshot/
project.domain.com/versions/1.0.0/
...

[Javier]

I did this before new dokka version which includes support for versioning, but I haven't tried it yet

[louiscad]

@Javier I read in many places that precompiled Gradle scripts were not working in all IDE versions. That's the reason I never used them to date, I don't want to add more problems. Do you know them to work reliably with IDE analysis starting from a given version of Android Studio? What about IDEA?

[Javier]

I use them in all IDEs since 2019

[louiscad]

So AS included?

[Javier]

For my projects I use IntelliJ IDEA usually, but in my work I use precompiled plugins every day in AS with no issues

[louiscad]

I guess I'll give it a try then, thank you for sharing your experience 🙂

[Javier]

I am going to commit the current state and add a new commit with a precompiled plugin too, so you can check them and the @Marcin Aman setup, up to you whatever you want to use later

[louiscad]

And thank you for trying the thing despite the big project size 🙂

[louiscad]

Still not sure how I would manage versioning though blob thinking fast
I want to embed KDocs (build from GFM format) on a website made with Material for MKDocs (the same that Dokka is using for its documentation website).

[Javier]

In my case, it is logic in Actions, dokka config is the same

[Javier]

And hope actions could use Kotlin instead yaml, JetBrains Space hasn't free unlimited minutes for open source projects :(

[louiscad]

I hate writing code in YAML for I think ovbious reasons, but I'd like to see an example for handling different versions of the doc

[louiscad]

I don't need unlimited minutes, I just need a reasonable limited amount of resources, and maybe more efficient build and doc tools 😜

[Javier]

I am going to share an article about my setup, hope soon, 24 hours days is blocking me :/

[louiscad]

Unlimited is never really unlimited. Time of our lives is limited for instance (and I'd rather not wait for builds for so long 😅)

[Javier]

I think the current limit is 2k minutes

[louiscad]

Your project is not open source?

[Javier]

If it fits your needs, you can mirror your GitHub repo in JetBrains spaced in seconds

[Javier]

Yeah, but because I was sharing the config for a few projects, I moved from copy pasting the config from one project to another to publishing my precompiled plugins, so maybe it is not so easy to see the config in that way

[louiscad]

I guess I'd be interested in it (JetBrains Space) for dev builds of the library.

[Javier]

I have good news for you

[Javier]

you can enable parallel

[louiscad]

This is a build of the entire project, not just Dokka?

[Javier]

In my case I am not using Jetbrains Space because I don't know if 2k mins is enough (I have a project which spend 40 min to build a thing)

[Javier]

Yeah, entire build, but cached

[Javier]

./gradlew build

[louiscad]

I'm wondering if it'd fail on a clean build

[Javier]

it can be strange, maybe on demand issue?

[Javier]

Nop, you haven't it

[louiscad]

@Javier Brought my calculator: 2K/40 = 50, so almost 2 builds per day if you work on the project 5 days a week.

[Javier]

Yeah, but that is a project, I spend a lot of time in CI when I try things

[louiscad]

Nop, you haven't it

What are you talking about?

[Javier]

org.gradle.configureondemand=true

[louiscad]

It's not enabled in Splitties

[Javier]

That option is giving me, sometimes, issues, so I enable/disable based on the project and the state of the project, but it isnt in your project, I am trying a gradlew clean build

[Javier]

From a clean build, it works

[Javier]

I had problems running the clean command, I have to do a gradlew --stop

[Javier]

But I don't know if it is related, I had this problem some times in other projects

[Javier]

I don't know enough your project about how you are publishing your artifacts

[louiscad]

How is artifacts publishing related?

[Javier]

Because in my config I extracted all the publishing related stuff to a plugin:

import com.javiersc.plugins.core.isSignificant
import org.jetbrains.dokka.gradle.DokkaTask

plugins {
    `maven-publish`
    signing
    id("org.jetbrains.dokka")
}

val docsJar by project.tasks.creating(Jar::class) {
    group = "build"
    description = "Assembles Javadoc jar file from for publishing"
    archiveClassifier.set("javadoc")
    dependsOn(tasks.named<DokkaTask>("dokkaHtml"))
}

configure<PublishingExtension> {
    publications {
        withType<MavenPublication> {
            pom {
                name.set(property("pomName").toString())
                description.set(property("pomDescription").toString())
                url.set(property("pomUrl").toString())

                licenses {
                    license {
                        name.set(property("pomLicenseName").toString())
                        url.set(property("pomLicenseUrl").toString())
                    }
                }

                developers {
                    developer {
                        id.set(property("pomDeveloperId").toString())
                        name.set(property("pomDeveloperName").toString())
                        email.set(property("pomDeveloperEmail").toString())
                    }
                }

                scm {
                    url.set(property("pomSmcUrl").toString())
                    connection.set(property("pomSmcConnection").toString())
                    developerConnection.set(property("pomSmcDeveloperConnection").toString())
                }
            }

            artifact(docsJar)
        }
    }
}

[louiscad]

In my case, I call a function defined in buildSrc

[Javier]

But as you can see there, that is the config for publishing a multiplatform library, but I am not applying there the multiplatform plugin, but in your case, you are setting things of the publication inside multiplatform functions

[Javier]

So I can't extract them easily

[Javier]

image.png

[Javier]

This one

[Javier]

So the precompiled plugin I want to create as sample, is really sad, because it is only:

plugins {
    maven-publish
    id("org.jetbrains.dokka")
}

[Javier]

Is there a reason to use afterEvaluate in some projects and in bitflags no?

[louiscad]

Re pre-compiled scripts: Aren't you impacted by this? https://youtrack.jetbrains.com/issue/KT-41142

[louiscad]

@Javier afterEvaluate is required modules that have and Android target.

[Javier]

I am trying without it publishing in local and it is working

[louiscad]

I'm pretty sure it's publishing with the wrong artifact names though

[Javier]

About that, I havent had that problem, there is a warning about kts is using different version that I was using

[Javier]

How can I check that?

[Javier]

image.png

[Javier]

I tried with activities

[Javier]

Is it correct?

[louiscad]

All published artifacts should start with splitties-, not just the directories

[Javier]

do you mean jar files?

[Javier]

image.png

[louiscad]

Also, the dependencies of one module on the other.

[louiscad]

You can try with views-dsl-material, it depends on several other ones.

[louiscad]

It might not be or no longer be needed for Kotlin Multiplatform projects 🤔

[Javier]

Do you think the projects that dont use afterEvaluate, will work with it?

[louiscad]

Is the .module file referencing the correct ones?

[louiscad]

If it was an Android library, I'm certain I'd need it: https://developer.android.com/studio/build/maven-publish-plugin

[louiscad]

But, anyway, I think it's possible to simply have 2 pre-compiled script plugins, with one for non Android targeting modules, and one for the Android targeting ones.

[Javier]

In this plugin I am only extracting only a few logic

[Javier]

image.png

[Javier]

I am going to push, if you want you can check if the modules are correct

[Javier]

https://github.com/LouisCAD/Splitties/pull/270/files

[Javier]

Some boilerplate is gone

[Kamil Doległo]

Can somebody please sum up the current issues? I tried to follow the thread, but I got lost.

1. Your PR on the wip-setup-dokka branch is almost OK @louiscad, you just have to make some small adjustments:
◦ change the configureEach<DokkaTask> to configureEach<DokkaTaskPartial> (and the relevant import as well)
◦ probably increase the JVM memory limit with something like org.gradle.jvmargs=-Xmx6g , because the compilation is failing (this project has a significant size after all)
◦ format your README.md s correctly for Dokka (see https://kotlinlang.org/docs/kotlin-doc.html#module-and-package-documentation)
2. There is no need to edit all buildfiles, everything works from the root one with allprojects block. There is no need to create a precompiled plugin too, you can however do that if you want to.

3. If you want to speedup the documentation you can reenable parallelism or just run Gradle with ./gradlew --parallel :dokkaHtmlMultiModule or whichever Dokka task you want.

4. I’m sorry to hear that you find the documentation ambiguous. Could you please tell me what is ambiguous for you and how would you like it to look?

[louiscad]

I need to check for the allProjects, and 3GB is enough (though I might experiment with more like 6GB once I understand what setting gigantic heap spaces implies, including machines with only 8GB or RAM and way less free if not swap) but yes, you got everything right.

About doc: There's a stuffed README, a website and a GitHub Wiki. It's not clear if one is abandoned and one is more a source of truth than the other. Ideally, the README would have a big link towards the website, and the FAQ and Wiki content would move to the website, or it would be mentioned from the website at a discoverable place.

Once that is cleared up, I'd like for the doc to simply show how to set it up for a Gradle project (no matter Android, Multiplatform or just JVM), mentioning settings to check to avoid bad onboarding experience (like HEAP size, Module.md files format), and have a less overwhelming presentation of the configuration options where you don't need to constantly scroll sideways to read every sentence, plus more clarity about the default values, so we know if it's something the default would not best fit for us.