Dokka 2 is really hungry for resources :wink: Had to set max heap size to 12G and close all other ap...

# dokka

[Robert Jaros]

Dokka 2 is really hungry for resources 😉 Had to set max heap size to 12G and close all other applications to generate documentation for my project with about 40 modules (It took 6 minutes with load average ~40 and made my 32GB thinkpad unresponsive).

[Robert Jaros]

It generated 676MB of html code.

[Oleg Yukhnevich]

In theory, Dokka 2.0.0 generator should use a bit less memory (at rendering stage) 🙂
But! Because DGPv2 now uses Gradle Workers, Dokka execution is highly parallel and spawns multiple processes...

Which heap you were increasing? Dokka execution or Gradle's?
There is also a Troubleshooting section which could help.
In addition to that, you can try to run DGP tasks with org.gradle.workers.max=2 to limit parallelism, but it will affect not only Dokka, so it's more of a last resort, if needed

BTW, is your project opensource?

[Robert Jaros]

Yes. https://github.com/rjaros/kilua
But I haven't pushed upgrade to Dokka 2 yet.

[Robert Jaros]

Which heap you were increasing? Dokka execution or Gradle's?

I have added:

dokkaGeneratorIsolation.set(ProcessIsolation {
    maxHeapSize.set("12g")
})

to all modules, and yes, it seems all modules are generated in parallel.

[Oleg Yukhnevich]

Does it fail with e.g 4g? if so, what is the exception?

[Oleg Yukhnevich]

Also, for projects with a lot of modules, ClassloaderIsolation might be a better choice regarding memory consumption - because it will not spawn separate processes, and so will use just Gradle heap

[Robert Jaros]

I'm doing some tests. I've changed to ClassloadedIsolation and with Gradle heap set to 6GB it fails after a few minutes with:

> Task :modules:kilua-trix:dokkaGenerateModuleHtml FAILED
The Daemon will expire after the build after running out of JVM heap space.
The project memory settings are likely not configured or are configured to an insufficient value.
The daemon will restart for the next build, which may increase subsequent build times.
These settings can be adjusted by setting 'org.gradle.jvmargs' in 'gradle.properties'.
The currently configured max heap space is '6 GiB' and the configured max metaspace is 'unknown'.

but there are still other tasks running, CPU at 100%, load avg ~10.

[Robert Jaros]

I have to kill this run, because it will probably not finish anytime soon.

[Robert Jaros]

Will try increasing heap size by 2GB until it works.

[Robert Jaros]

java.lang.OutOfMemoryError: Java heap space on 8GB

[Robert Jaros]

Lost patience after 17 minutes run with 10GB. No exceptions but no results either.

[Robert Jaros]

Same with 12GB. I'm afraid there is some kind of deadlock there, because I can't even kill the java daemon process without -9 signal.

[Robert Jaros]

I will try with 16GB once more

[Robert Jaros]

Still doesn't work

[Oleg Yukhnevich]

I will take a look on your project a bit later
Could you publish a branch where you were able to generate documentaion?

[Robert Jaros]

I'll let you know

[Robert Jaros]

I've reverted to ProcessIsolation and it works with 8g.

[Robert Jaros]

I've pushed my changes. Current Dokka configuration is in the `buildSrc`:
https://github.com/rjaros/kilua/blob/main/buildSrc/src/main/kotlin/Shared.kt#L207

[Oleg Yukhnevich]

Okay, so I took a look on your project.
And yes, Dokka needs rather big amount of memory to process one of the modules: kilua module - it's just big
Previously Dokka were executed inside Gradle daemon and wasn't able to run in parallel in most cases - and so 6GB (in gradle.properties) were enough to generate documentation for every module separately
That's why now with ClassloadedIsolation you are getting OOM with the same or even bigger memory there - because Dokka run in parallel

So yes, for your use-case it's better to use ProcessIsolation with 8GB because of this big module. You could also set those 8GB only for that kilua module - may be it will use less memory overall (by default we set 2GB for worker)

Also, just keeping you informed: thanks to your project, I found some places which could optimize Dokka memory consumption 🙂

In any case, thanks for trying DGPv2!

[Robert Jaros]

Thank you for taking care of my case! 🙂