https://kotlinlang.org logo
#getting-started
Title
# getting-started
i

igor.wojda

03/12/2024, 10:36 PM
Hey fellow devs, I'm reaching out for some assistance and hoping to tap into the collective wisdom of this community. Context: I'm the creator of Konsist, a new tool designed for structural linting, which works by parsing Kotlin files to offer a developer-friendly API for creating custom checks. The test run is crashing with when parsing a large number of classes - reason
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
error. Despite my attempts to address this (including tweaking
gradle.properties
flags). I suspect my implementation is not-optimal, but I am not sure on how to find the root cause of the problem (error message is not helpful). I've prepared a complete, runnable example that includes: • The source code for
Konsist 0.14.0
(directly in the
main
source set for easier debugging rather than as a separate dependency) • A directory (
parsedClasses
) containing classes for Konsist to parse (and used them in test) • A sample test class (
SampleKonsistTest
) that triggers the parsing of classes in
parsedClasses
To reproduce the issue, you can simply download attached project and execute the following command:
./gradlew --stop && ./gradlew konsistTest --rerun-tasks
. I'm really hoping someone with experience in troubleshooting memory-related issues can shed some light on this for me. Any guidance or suggestions would be greatly appreciated 🙏
a

Adam S

03/12/2024, 11:04 PM
hi 👋 Do you have IntelliJ Ultimate? If so, you can try profiling the task
Here's the flamegraph for memory usage
so it looks like it's trying to load waaaaay too many heavy Psi classes - does it really need to load them all simultaneously, or can Konsist analyse each Psi one-by-one?
i

igor.wojda

03/12/2024, 11:12 PM
Konsist needs to parse these files to have in-memory representation of the codebase. Ideally this should happen at start, as some checks may require all classes to perform verification eventually. 10K of files does not seem like a lot 🤔 It is surprising that ~37MB of files after parsing takes more then few gigabytes of memory 🤔
a

Adam S

03/12/2024, 11:15 PM
it's not necessarily the content of the files, it's that the Psi classes contain a lot of data
🤔 1
typically the Psi classes are visited, not mapped into a huge list :) https://plugins.jetbrains.com/docs/intellij/navigating-psi.html#bottom-up-navigation
👍 1
s

Stefan Oltmann

03/13/2024, 6:17 AM
If you don’t have IDEA Ultimate I think you are eligible for a free license. 😉
a

Adam S

03/13/2024, 1:12 PM
I had to look into some memory issues recently, and I found a really good article about the tools that were available for JVM memory inspection. And I can't find it!
Looking more through the Konsist code, I think it'd be a good idea to replace all of the Lists with Sequences, so the results can be evaluated lazily
I also saw this function https://github.com/LemonAppDev/konsist/blob/81324e09fe46c9f1095b133713ae0062e742c2e3/lib/src/main/kotlin/com/lemonappdev/konsist/core/provider/util/KoDeclarationProviderCoreUtil.kt#L147-L169 It looks like localDeclarations is going to have nestedDeclarations added to it multiple times, and so there are going to be many many many duplicates
I had to look into some memory issues recently, and I found a really good article about the tools that were available for JVM memory inspection. And I can't find it!
I found it! https://krzysztofslusarski.github.io/2022/12/12/async-manual.html