Hey fellow devs, I'm reaching out for some assista...

# getting-started

[igor.wojda]

Hey fellow devs,
I'm reaching out for some assistance and hoping to tap into the collective wisdom of this community.

Context: I'm the creator of Konsist, a new tool designed for structural linting, which works by parsing Kotlin files to offer a developer-friendly API for creating custom checks.

The test run is crashing with when parsing a large number of classes - reason java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects error. Despite my attempts to address this (including tweaking gradle.properties flags). I suspect my implementation is not-optimal, but I am not sure on how to find the root cause of the problem (error message is not helpful).

I've prepared a complete, runnable example that includes:
• The source code for Konsist 0.14.0 (directly in the main source set for easier debugging rather than as a separate dependency)
• A directory (parsedClasses) containing classes for Konsist to parse (and used them in test)
• A sample test class (SampleKonsistTest) that triggers the parsing of classes in parsedClasses
To reproduce the issue, you can simply download attached project and execute the following command: ./gradlew --stop && ./gradlew konsistTest --rerun-tasks.

I'm really hoping someone with experience in troubleshooting memory-related issues can shed some light on this for me. Any guidance or suggestions would be greatly appreciated 🙏

memory-test.zipstackTrace.log

[Adam S]

hi 👋

Do you have IntelliJ Ultimate? If so, you can try profiling the task

[Adam S]

Here's the flamegraph for memory usage

[Adam S]

so it looks like it's trying to load waaaaay too many heavy Psi classes - does it really need to load them all simultaneously, or can Konsist analyse each Psi one-by-one?

[igor.wojda]

Konsist needs to parse these files to have in-memory representation of the codebase. Ideally this should happen at start, as some checks may require all classes to perform verification eventually. 10K of files does not seem like a lot 🤔 It is surprising that ~37MB of files after parsing takes more then few gigabytes of memory 🤔

[Adam S]

it's not necessarily the content of the files, it's that the Psi classes contain a lot of data

[Adam S]

typically the Psi classes are visited, not mapped into a huge list :)

https://plugins.jetbrains.com/docs/intellij/navigating-psi.html#bottom-up-navigation

[Stefan Oltmann]

If you don’t have IDEA Ultimate I think you are eligible for a free license. 😉

[Adam S]

I had to look into some memory issues recently, and I found a really good article about the tools that were available for JVM memory inspection. And I can't find it!

[Adam S]

Looking more through the Konsist code, I think it'd be a good idea to replace all of the Lists with Sequences, so the results can be evaluated lazily

[Adam S]

I also saw this function https://github.com/LemonAppDev/konsist/blob/81324e09fe46c9f1095b133713ae0062e742c2e3/lib/src/main/kotlin/com/lemonappdev/konsist/core/provider/util/KoDeclarationProviderCoreUtil.kt#L147-L169

It looks like localDeclarations is going to have nestedDeclarations added to it multiple times, and so there are going to be many many many duplicates

[Adam S]

I had to look into some memory issues recently, and I found a really good article about the tools that were available for JVM memory inspection. And I can't find it!

I found it! https://krzysztofslusarski.github.io/2022/12/12/async-manual.html