Hey! I’ve been debugging remote cache misses in ou...

# gradle

[egorand]

Hey! I’ve been debugging remote cache misses in our project, and noticed that a compileDebugKotlin build performed by CI that populated the cache and a local build that consumed the cache print out different task class names, which seemingly results in a miss - CI says it ran KotlinCompileWithWorkers and the dev machine says it ran KotlinCompile (see the screenshot). Upon some investigation I landed on this code that seems to pick a different task based on whether build parallelization is enabled (controlled by kotlin.parallel.tasks.in.project before 1.5.20 and IIUC by org.gradle.parallel after that - please correct me if I’m wrong):

https://github.com/JetBrains/kotlin/blob/v1.5.31/libraries/tools/kotlin-gradle-plugin/src/main/kotlin/org/jetbrains/kotlin/gradle/tasks/TasksProvider.kt#L165-L166

I’m not sure if this is a red herring and our cache misses have a different cause, but I’m curious if it’s a known issue. I also noticed that the following commit removes the KotlinCompileWithWorkers task, which seemingly will ship in 1.6.20:

https://github.com/JetBrains/kotlin/commit/904f9c72f2a8996511def90d76e054a2f9f98b0e

So far we’re planning to enable parallelization for our CI builds, but would love to know more about this issue.

[tapchicoma]

hi, it is known issue. Better to use on CI and locally either both --parallel Gradle flag or same value for kotlin.parallel.tasks.in.project.

1.6.20 removes kotlin.parallel.tasks.in.project flag and fixes this issue

[egorand]

Thanks! Has this issue been filed to YouTrack, and if yes - do you have the link?

[tapchicoma]

nope, it wasn't filed into YouTrack and we found it during internal investigation. It was fixed as a part of this issue.

[tapchicoma]

ah yeah - one downside of enabling kotlin.paralle.tasks.in.project - after a build failure - next build will be non-incremental for the module where failure was. It was also fixed.

[egorand]

I see, which version will the fix be included in - 1.6.20? Also, looks like we’d need to enable org.gradle.parallel since kotlin.paralle.tasks.in.project has been deprecated in 1.5.20?

[tapchicoma]

first question - yes, second one - you should have both <http://kotlin.parallel.tasks.in|kotlin.parallel.tasks.in>.project=false and org.gradle.parallel=true to mitigate remote cache/incremental compilation issues, but have "parallel" build.

Note that org.gradle.parallel flag only affect whether Gradle will build different modules in the project in parallel. But Gradle could still execute tasks in parallel inside the module, which is controlled by org.gradle.workers.max property

[egorand]

Thanks, this is very helpful! I don’t think we’re overriding kotlin.parallel.tasks.in.project on either CI or dev machines, so curious why we’re seeing different tasks invoked - is that an effect of org.gradle.parallel being set to different values? In that case, do we need to set <http://kotlin.parallel.tasks.in|kotlin.parallel.tasks.in>.project=false, or just make sure we set org.gradle.parallel to the same value in both setups?

[tapchicoma]

is that an effect of org.gradle.parallel being set to different values?

Yes, alongside with deprecation kotlin.parallel.tasks.in.project - default value (if value was not set explicetly) set to depend on --parallel switch. I must admit that was a mistake and it actually should rather depend on org.gradle.workers.max value.

So if you will set kotlin.parallel.tasks.in.project explicitly - this default value should be overridden by explicit one.

[egorand]

Hmm, what I’m seeing on my local machine with <http://kotlin.parallel.tasks.in|kotlin.parallel.tasks.in>.project=false is that for some reason compileKotlin tasks for different Gradle modules are executed sequentially on a single Gradle worker, while all the others stay IDLE. I’m going to try removing kotlin.parallel.tasks.in.project override for both our local and CI setups to get the setting to just depend on org.gradle.parallel - I believe that’s the behavior we want.

[tapchicoma]

are executed sequentially on a single Gradle worker

There were 2 type of tasks - with compilation via worker (left) or with compilation via task action (removed, <http://kotlin.parallel.tasks.in|kotlin.parallel.tasks.in>.project=false). The second one, indeed, most probably will be executed sequentially