https://kotlinlang.org logo
Join SlackCommunities
Powered by
Is there away to not recompile `olderVersionsDir`e...
# dokka
a
Is there away to not recompile `olderVersionsDir`every time we run 
DokkaMultiModuleTask
. The reason being that we do release once per month (sometimes even more frequent) and the folders are piling up and and it takes longer and longer time to recompile/generate HTML for all old version. Maybe we are doing some things wrong? 🤔
👀 1
This is how our config looks like. Still running Dokka “1.9.20”
Copy code
// Dokka
    val dokkaVersion: String by rootProject.extra
    val currentVersion = evaluateVersionName()
    val currentYear = Calendar.getInstance().get(Calendar.YEAR)
    pluginManager.withPlugin("org.jetbrains.dokka") {
        afterEvaluate {
            val dokkaPlugin by configurations
            dependencies {
                dokkaPlugin("org.jetbrains.dokka:android-documentation-plugin:$dokkaVersion")
                dokkaPlugin("org.jetbrains.dokka:versioning-plugin:$dokkaVersion")
            }
            this.tasks.withType(DokkaMultiModuleTask::class) {
                pluginConfiguration<VersioningPlugin, VersioningConfiguration> {
                    version = currentVersion
                    olderVersionsDir = file("documentation/oldVersions")
                    renderVersionsNavigationOnAllPages = false
                }
                pluginConfiguration<DokkaBase, DokkaBaseConfiguration> {
                    customAssets = listOf(file("documentation/logo-icon.svg"))
                    footerMessage = "© X Systems 1999-$currentYear"
                }
                outputDirectory.set(file("documentation/html"))
                includes.from("documentation/packages.md")
            }

            tasks.register("updateReadme", Exec::class) {
                commandLine("./../../scripts/update_readme.sh")
            }

            // Always copy Readme.md before running Dokka generation.
            tasks.findByName("dokkaHtmlPartial")?.dependsOn("updateReadme")
        }
    }
e
we just disabled the versioning plugin because it was getting so bad
a
So you basically only present the latest version now?
yes black 1
o
Hey! Could you expand a bit on:
Is there away to not recompile `olderVersionsDir`every time
Dokka mostly just copies what is in older dir And yes, the output will be bigger and bigger, as currently it's only possible to include FULL HTML of previous versions inside the final bundle
a
Yes it is mostly copying, but lets say that we have 15 older releases and each release/version contains 3000 files then we have 45 000 files so it takes a while to generate/copy. Even my computer struggles to do this (Macbook Pro M1 Pro, approx 20-25 min) not to mention github runners.
o
yeah, that's true, and from the fast check I don't see a strait-forward way to improve this with current architecture The code for this is here BTW, may be you will be able to come-up with something - contributions are welcome!
👍 1
a
Thanks. However currently we are experiencing problems with this bug and have spend quite some time to try to release our latest version. https://github.com/Kotlin/dokka/issues/3606
What used to take 20-25 min to build now takes several hours. 😢
o
Do you have a way to reproduce the issue locally? It's hard to investigate this issue without the reproducer As an idea, you could try to reduce parallelism for the JVM which executes Dokka Or you could try to update to Dokka 2.0.0, there were some performance improvements there which could affect this
a
This but is now not intermittent in my team, it occurs every time. All (3) android devs have the same issue. It works however without issues on Github runners.
If you are referring to
Copy code
org.gradle.parallel=true
we are not even using this.
We have just managed to get our release out but I think we might look into migrating to Dokka 2.0
o
If you are referring to
No, I'm about JVM cores limitation, f.e. like this (SO):
Copy code
-XX:ActiveProcessorCount=2 # the value could be different
This could of course affect the performance, but that's probably why on GitHub runners, there are no issues - less cores and so less possibility to receive this specific deadlock
a
What is strange is that it worked just fine 1-2 months ago. I cannot find anything that might have causes the builds slowing down like this.,.,.
Is there a good way to enable a more verbose mode that can give you some valueable data?
o
it could be anything TBH, as we don't fully understand the root cause of the issue - may be now you just have more things to document
more verbose mode
nope, there is nothing specific here which could help except for access to codebase and running checks with profiler/debugger to understand the root cause
👍 1
a
Yes, I am now just been trying different things that might impact the builds. What I get so far is that the current (latest) version of the documentation takes 8-10 min which is not that strange. I think that is roughly what it use to take on my computer. The odd thing is that the copying of the oldversions seem to take a really long time. Almost like the computer is struggling with copying the files between folders…
Hmm, my issue might be cause by Microsoft Defender which might have gotten more aggressive in scanning files. 🤔 I managed to temporarily exclude a folder and cloned my repos there and ran Dokka build task there. Initial investigation seems that build time drastically improved.
🎉 1
Is there a “Multi-module example project” using Dokka2? I have gone through the Dokka2 migration guide and it is pretty confusing and does not map at all to our setup. 🤔 https://kotlinlang.org/docs/dokka-migration.html#migrate-your-project I already posted the project level setup above but here is a more cleaned up version of it.
Copy code
import org.jetbrains.dokka.base.DokkaBase
import org.jetbrains.dokka.base.DokkaBaseConfiguration
import org.jetbrains.dokka.gradle.DokkaMultiModuleTask
import org.jetbrains.dokka.versioning.VersioningConfiguration
import org.jetbrains.dokka.versioning.VersioningPlugin    

/////
<code>
/////
pluginManager.withPlugin("org.jetbrains.dokka") {
        afterEvaluate {
            val dokkaPlugin by configurations
            dependencies {
                dokkaPlugin("org.jetbrains.dokka:android-documentation-plugin:$dokkaVersion")
                dokkaPlugin("org.jetbrains.dokka:versioning-plugin:$dokkaVersion")
            }
            this.tasks.withType(DokkaMultiModuleTask::class) {
                pluginConfiguration<VersioningPlugin, VersioningConfiguration> {
                    version = currentVersion
                    olderVersionsDir = file("documentation/oldVersions")
                    renderVersionsNavigationOnAllPages = false
                }
                pluginConfiguration<DokkaBase, DokkaBaseConfiguration> {
                    customAssets = listOf(file("documentation/logo-icon.svg"))
                    footerMessage = "© Company 1999-$currentYear"
                }
                outputDirectory.set(file("documentation/html"))
                includes.from("documentation/packages.md")
            }
        }
    }
So this is an android multimodule project which has some modules that should be part of Dokka and some that are excluded, hence the need for 
this.tasks.withType(DokkaMultiModuleTask::class)
How would you suggest me to proceed?
o
yeah, sure, we have an example in the repo
it is pretty confusing and does not map at all to our setup.
yeah, Gradle allows to much flexibility, and so it can be done in a million ways... Still, the final idea with convention plugins is the recommended way Dokka 2.0.0 with DGPv2
which has some modules that should be part of Dokka and some that are excluded
with DGPv2 you explicitly declare which projects will be aggregated, so you can just omit projects which should not be documented (here is how projects are included in an example)
a
Thanks for the pointers. I got a few things working and hopefully most things work with Dokka 2.0.0 with DGPv2 now. What I got stuck on currently is the olderVersions
Copy code
pluginsConfiguration.versioning {
    olderVersionsDir.set(layout.projectDirectory.dir("documentation/oldVersions")) // Tying to access subProjects folders and fails bc the directory paths resolves incorrectly
}
o
not sure what path you want here, but it's just a path, so if you need to access parent dir you can write there 
"../documentation/oldVersions"
, or use other Gradle APIs
a
I got it to work. 🙂 Root project: (there might be some cleanup to handle even though I tried to remove unused code)
Copy code
dependencies {
    dokka(project(":foundation:"))
    dokka(project(":design:"))
    dokka(project(":icons:"))
}

pluginManager.withPlugin("org.jetbrains.dokka") {
    dokka {
        pluginsConfiguration.versioning {
            olderVersionsDir.set(layout.projectDirectory.dir("documentation/oldVersions"))
        }
    }
}

allprojects {
    repositories {
        google()
        mavenCentral()
        maven("<https://jitpack.io>")
        maven("<https://plugins.gradle.org/m2/>")
    }

    /* other code */

    // Dokka
    val dokkaVersion: String by rootProject.extra
    val currentVersion = evaluateVersionName()
    val currentYear = Calendar.getInstance().get(Calendar.YEAR)
    pluginManager.withPlugin("org.jetbrains.dokka") {
        dokka {
            dokkaSourceSets.configureEach {
                includes.from("$projectDir/README.md")
                sourceLink {
                    basePublicationsDirectory
                    localDirectory.set(file("src/main/java"))
                    remoteUrl("<https://android.some-url.net/src>")
                    reportUndocumented.set(true)
                    skipEmptyPackages.set(true)
                }
            }

            dokkaPublications.html {
                outputDirectory.set(layout.projectDirectory.dir("documentation/html"))
                includes.from("documentation/packages.md")
            }

            pluginsConfiguration.html {
                customAssets.from("${rootProject.rootDir}/documentation/logo-icon.svg")
                footerMessage.set("© Some company 1999-$currentYear")
            }

            pluginsConfiguration.versioning {
                version = currentVersion
                renderVersionsNavigationOnAllPages = false
            }
        }
    }
}
s
honestly I think that it would be really nice if there were a built-in solution that will download the 
maven-metadata.xml
for a specific group-artifact pair, as in this file there is a list of every* single version for an artifact. (As an example, a project were being published to 
ca.solo-studios:slf4k:1.2.3
, then the 
maven-metadata.xml
could be downloaded from 
<https://repo1.maven.org/maven2/ca/solo-studios/slf4k/maven-metadata.xml>
) The metadata would then be used to recursively download all the 
-javadoc
jars for a project before unpacking each of them into the directory for the previous versions. *technically, it will not include every version. if 
-SNAPSHOT
versions are published, then it will only be listed once even though the same 
-SNAPSHOT
version can be re-published multiple times and gradle will publish it under the same directory with a new timestamp and an increasing number appended to the end (eg. you might publish 
ca/solo-studios/slf4k/0.5.4-SNAPSHOT/slf4k-0.5.4-20240922.223728-1.jar
and the next time you might publish 
ca/solo-studios/slf4k/0.5.4-SNAPSHOT/slf4k-0.5.4-20240922.224020-2.jar
. these would only be listed once, under 
0.5.4-SNAPSHOT
.)
a
How is this related to this 🧵 @solonovamax? Or am I missing anything 😅
s
it's related to the issue of having a ton of garbage building up in the 
olderVersionsDir
that can easily just be recreated bc it's already published to maven I meant to say it back when the thread was originally posted but forgot to
o
Feel free to leave your comments here: https://github.com/Kotlin/dokka/issues/3977 but the main "issue" here is that there is no multi-module aggregated artifact, and just single-modules currently can not be aggregated in the same way, as multi-module aggregate is created from so called 
partial
outputs so it's a bit harder then that 😞
👍 1
5 Views
Open in Slack
PreviousNext
Powered by