is there a way to specify how much memory the kotlin compiler should use in the KotlinCompile block?...

# gradle

[janvladimirmostert]

is there a way to specify how much memory the kotlin compiler should use in the KotlinCompile block?

tasks.withType<KotlinCompile> {
	incremental = true
	kotlinOptions.jvmTarget = "1.8"
}

builds that usually took 1 minute are now taking 7-15 minutes, not sure if it's due to the code-base growing pass a certain point that's now causing this slowdown, new intellij version, new kotlin version or any of several dozen permutations of things that could have changed.
i want to see if giving the compiler more memory will speed up compile times or get it back to "normal"

[gildor]

you should use gradle.properties to set jvm heap size

[gildor]

https://docs.gradle.org/current/userguide/build_environment.html#sec:configuring_jvm_memory

[janvladimirmostert]

awesome, thanks Andrey!

[gildor]

There are also some other Kotlin Daemon properties related to memory, but never used them

[gildor]

also, check Gradle build scan info

[gildor]

if it’s memory problem, you will see GC time there

[janvladimirmostert]

i've played around with various settings, maven takes 7 minutes to build the whole project, gradle seems to get stuck at compileKotlin and then after 25 minutes, i kill it.

I've done the build scan, and it gave suggestions to add some extra params in the gradle.properties file:

org.gradle.caching=true
#org.gradle.console=rich
org.gradle.console=verbose
org.gradle.daemon=true
org.gradle.configureondemand=true
org.gradle.jvmargs=-Xmx4g -XX:MaxMetaspaceSize=1024m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
org.gradle.parallel=true

kotlin.incremental=true

running --scan doesn't do anything if the build doesn't complete.

This is a mixed Java + Kotlin project (only two or three Java classes), all the Java sources are in src/main/java and Kotlin sources are in src/main/kotlin

import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
import org.jetbrains.kotlin.daemon.common.configureDaemonJVMOptions
import org.jetbrains.kotlin.gradle.plugin.KotlinSourceSet
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

group = "<http://za.co|za.co>. ......"
version = "1.2.0"
description = " ......"

java.sourceCompatibility = JavaVersion.VERSION_1_8
java.targetCompatibility = JavaVersion.VERSION_1_8

application {
	mainClassName = " ......"
}

plugins {

	val kotlinVersion = "1.3.30"

	application
	kotlin("jvm").version(kotlinVersion )
	java
	maven
	id("com.github.johnrengelman.shadow").version("2.0.4")

	id("org.jetbrains.kotlin.plugin.noarg").version(kotlinVersion)
}

tasks.withType<ShadowJar> {
	baseName = "app"
}

repositories {
	mavenLocal()
	jcenter()
	maven("<http://repo.maven.apache.org/maven2>")
}

tasks.withType<Jar> {
	manifest {
		attributes(mapOf(
			"Main-Class" to " ......",
			"mainClassName" to " ......"
		))
	}
}

apply {
	plugin("org.jetbrains.kotlin.plugin.noarg")
}

tasks.withType<KotlinCompile> {
	incremental = true
	kotlinOptions.jvmTarget = "1.8"
}

dependencies {
 ...
}

[janvladimirmostert]

running it with gradle build --debug --warning-mode=all --full-stacktrace --console=verbose now

seeing lots of

13:59:51.497 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
13:59:53.695 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting OS memory status event {Total: 17179869184, Free: 5305978880}
13:59:53.695 [DEBUG] [org.gradle.launcher.daemon.server.health.LowMemoryDaemonExpirationStrategy] Received memory status update: {Total: 17179869184, Free: 5305978880}
13:59:53.695 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting JVM memory status event {Maximum: 7635730432, Committed: 487587840}

[janvladimirmostert]

got it working, had to remove the java and shadow plugin
now it's running fine

[janvladimirmostert]

maven=7min
gradle=2min

huge difference 😮

[janvladimirmostert]

58 seconds after a bit more tweaking, wow!

[gildor]

Removed java?

[gildor]

Yes, it looks as some memory issue

[gildor]

Why do you set manifest properties manually if you already have shadow plugin?
Maybe if you have reproducible example you could report an issue, today I discussed very similar issue

[janvladimirmostert]

i'll try and reproduce it, not clued up enough on gradle to know if what i'm doing is completely and utterly wrong or if i was looking at a gradle bug, still getting used to gradle

[janvladimirmostert]

Removed java?

i had java and kotlin under plugins, seems that the kotlin plugin compiles java too, so no need for that. Not sure if the kotlin and java plugins are locking resources or something

[gildor]

Yes, Kotlin uses Java plugin implicitly

[gildor]

So it shouldn't be any conflict

[gildor]

So you know on which task it was so slow?

[janvladimirmostert]

so maybe it's the shadow plugin then

[janvladimirmostert]

blocking compileKotlin for some reason

[janvladimirmostert]

id("com.github.johnrengelman.shadow").version("2.0.4")

[janvladimirmostert]

let me put it back and see if it hangs again

[gildor]

This issue with build time which I mentioned above
https://kotlinlang.slack.com/archives/C19FD9681/p1556087653127300?thread_ts=1556046673.121500&cid=C19FD9681

[gildor]

Why do you use so old version of shadow plugin? There is version 5.0 available
https://plugins.gradle.org/plugin/com.github.johnrengelman.shadow

[janvladimirmostert]

ah, that's probably a contributing factor, let me try 5.0

[janvladimirmostert]

2.0.4 is showed on ktor website under normal gradle script, but 5.0.0 is shown under kotlin dsl
https://ktor.io/servers/deploy/packing/fatjar.html

[janvladimirmostert]

probably where i got it from

[janvladimirmostert]

message has been deleted

[janvladimirmostert]

message has been deleted

[snowe]

did you figure out if it was shadow? We don't use the shadow plugin, but we do have java and kotlin plugins applied, and our gradle build is taking over 45 minutes, while the maven build only takes about 10...

[janvladimirmostert]

going to shadowJar V5.0.0 fixes the build times, but i'm not getting an executable jar, it's not putting the Main-Class in the manifest

was expecting application plugin to do that

application {
	mainClassName = "AppApi"
}

or even this:

tasks.withType<ShadowJar> {
	manifest.attributes.put("Main-Class", application.mainClassName)
}

but there's no "Main-Class" in the manifest, so yes, it actually finishes building, but the shadow jar doesn't work at all

[janvladimirmostert]

doing this:

tasks.withType<Jar> {
	manifest.attributes.put("Main-Class", application.mainClassName)
}

gives a JNI error

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: kotlin/TypeCastException
	at java.lang.Class.getDeclaredMethods0(Native Method)

[gildor]

Looks like a bug, 5.0 is the latest version and should be used with Gradle 5.x

[janvladimirmostert]

comparing maven output

[janvladimirmostert]

to gradle output

[janvladimirmostert]

i notice that gradle build still has Java and Kotlin directories instead of merging classes ...

[janvladimirmostert]

does it even when i have sourceSets in place:

sourceSets.main {
	java.srcDir("src/main/java")
	withConvention(KotlinSourceSet::class) {
		kotlin.srcDir("src/main/kotlin")
	}
}

[janvladimirmostert]

@snowe i'm still having insanely long build times in intellij even though it's quick (and without shadow working) on command line

[gildor]

Yes, this is how compilation in new version of Gradle work, separate dirs for classes of Kotlin and Java

[gildor]

Also don't think that you need this source sets config, you can use those dirs by default

And do you have any particular reasons to keep your Java and Kotlin sources separately?

[janvladimirmostert]

The Java code is just 3 enormous and messy classes that somebody else wrote that is in a completely different package, that code is not going to be maintained by anyone at all, so it's kinda abandoned in the java folder and wrapped by a library in the kotlin folder

I've removed the sourceSets, still not getting an executable jar, manually adding a manifest in tasks.withType<Jar> generates the correct manifest, although i get a JNI error on startup.

Busy upgrading my gradle version now from 5.1 to 5.4 to see if that fixes it.

[gildor]

I’ve removed the sourceSets, still not getting an executable jar

Which task are you use? To make execcutable jar with all dependencies you need application plugin + shadow plugin

[janvladimirmostert]

This is my full build.gradle.kts with dependencies cut off to fit in here:

import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

group = "za.co.convirt"
version = "1.2.0"
description = "App Build"

java.sourceCompatibility = JavaVersion.VERSION_1_8
java.targetCompatibility = JavaVersion.VERSION_1_8

application {
	mainClassName = "AppBuildApi"
}

plugins {

	val kotlinVersion = "1.3.30"

	application

	id("org.jetbrains.kotlin.plugin.noarg").version(kotlinVersion)
	kotlin("jvm").version(kotlinVersion)
	id("com.github.johnrengelman.shadow").version("5.0.0")
	maven

}

repositories {
	mavenLocal()
	jcenter()
	maven("<http://repo.maven.apache.org/maven2>")
}

tasks.withType<Jar> {
	manifest {
		attributes.put("Main-Class", application.mainClassName)
	}
}

tasks.withType<KotlinCompile> {
	incremental = true
	kotlinOptions.jvmTarget = "1.8"
}

dependencies {

	compile(kotlin("stdlib-jdk8"))
	compile(kotlin("reflect"))
	compile(kotlin("test-junit")

[janvladimirmostert]

i've got both application and shadow plugin

[gildor]

now use shadowJar task to build executable shadow jar

[gildor]

and you can remove manifest block

[janvladimirmostert]

mmm, this is something to get used to, in Maven, everything is tied to a build cycle. Let me give that a try ...

[gildor]

just by default shadow plugin doesn’t replace default jar task, but adds additional shadowJar that allows you to build shadow version. Because non shadow is faster to build, so it good to use during development, but it’s easy to disable default jar task and add shadowJar instead to default build lifecycle, if you want

[janvladimirmostert]

Execution failed for task ':shadowJar'.
> shadow.org.apache.tools.zip.Zip64RequiredException: archive contains more than 65535 entries.

will probably need that zip64 setting here

[gildor]

yeah, correct

[gildor]

are you sure that you need compile(kotlin("test-junit") instead of testCompile? now you bundling all your test infrastructure with your app

[janvladimirmostert]

awesome, after adding

tasks.withType<ShadowJar> {
	isZip64 = true
}

i've got a running app, thanks for the help Andrey!

[janvladimirmostert]

you're correct, that should be testCompile

[gildor]

actually even better to use testImplementation and implementation instead of compile.
Compile configuration is deprecated

[gildor]

tasks.withType<ShadowJar> {

Probably you can just use static accessor for this task:

tasks.shadowJar {}

[janvladimirmostert]

is there a difference between static accessor and the withType syntax ?

[snowe]

isn't .withType the static accessor and .shadowJar is dynamic accessor?

[gildor]

no, withType applies settings to all task with type ShadowJar

[snowe]

what does that have to do with being static.

[gildor]

sorry, I mean type safe accessors, not static

[gildor]

tasks.shadowJar or tasks { shadowJar {} } is type safe accessor generated by Kotlin DSL, so you can just access any task created by all applied plugins

[gildor]

so no need to know task type, no need to configure all task useing withType, you configure only what you need and have code completion

[snowe]

Hmm. I still don't see how withType forces you to configure more than the type-safe version does. I use withType everywhere I can, because IntelliJ is much faster at resolving the types.

[gildor]

because you know that this task exist

[gildor]

tasks.withType<ShadowJar> will configure nothing if task of this type doesn’t exist in current module

[gildor]

IntelliJ is much faster at resolving the types.

Not sure what you mean

[gildor]

As you can see you not only have type and only what you should know is name of task, but also have auto complete for all other available tasks

[gildor]

I don’t want to say that tasks.withType is somehow bad, but usually it’s not necessary to configure all tasks of this type, you need only one particular task, but also no need to know type of task, just name

[snowe]

tasks.withType<ShadowJar> will configure nothing if task of this type doesn’t exist in current module (edited)

hmm. I see what you mean now. In regards to intellij, it takes about 10-15 seconds for the appropriate type-safe accessor to pop up, while it only takes a second or two for the class name in withType<> to pop up.

[gildor]

works for me relatively fast, but not too much fast 😀

[gildor]

I agree, that tooling is still not perfect, sometimes painfully slow, especially with new plugins/scripts, but Kotlin DSL itself is quite good last few versions

[janvladimirmostert]

it has improved a lot, i tried it with Gradle 4.10 and it was horrible breaking more than it was working. With 5.4 now it seems to be fairly solid, not perfect yet

[gildor]

oh yeah, believe me, I play with it starting from first public version 0.1 😀
I didn’t think that it take for so long, but clean up all this dynamic approach from Gradle ecosystem is huge work

[janvladimirmostert]

i was happy to stick with Maven, kotlin multiplatform seems to work better with gradle, so had to make the switch. and writing groovy was horrible, so decided to give the Kotlin dsl a try

[gildor]

and yeah, tooling and kotlin scripting are big problems, even now not perfect