I started seeing this message after updating Gradle to 8 0 r kotlinlang #gradle

I started seeing this message after updating Gradl...

eygraber

01/19/2023, 9:44 PM

I started seeing this message after updating Gradle to 8.0-rc-1 and Kotlin to 1.8 (and setting

kotlin.daemon.useFallbackStrategy=false

). Is this talking about the Kotlin compiler daemon or the Gradle daemon? Bonus points if anyone knows what the issue could be 😅

Copy code

Unable to clear jar cache after compilation, maybe daemon is already down: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
	java.net.ConnectException: Connection refused

eygraber

01/19/2023, 9:44 PM

There are no errors in the build. Tasks are executing, and suddenly that message appears and the build fails.

Vampire

01/19/2023, 9:48 PM

It's the Kotlin compile daemon that is meant.

Vampire

01/19/2023, 9:50 PM

Maybe look at the Kotlin compile daemon logs if there are any to see whether there is something suspicious, or maybe the system logs to see whether it got killed for example by OOM killer.

eygraber

01/19/2023, 9:50 PM

I don't think I can access those logs because it is on GitHub Actions

eygraber

01/19/2023, 9:51 PM

But I'm assuming it's the OOM killer and the fallback strategy has been masking that for a while

Vampire

01/19/2023, 9:55 PM

Add the https://github.com/marketplace/actions/debugging-with-ssh action to be able to SSH into the machine, and add a step that runs

if: always()

after the failing step and keeps the workflow running in case the debugging-with-ssh action does not do that already anyway.

eygraber

01/19/2023, 9:58 PM

Oh cool, I'll check that out thanks

tapchicoma

01/19/2023, 10:51 PM

yeah, looks like OOM in Kotlin daemon. Please check the Kotlin daemon logs inside

/tmp

directory on the runner

eygraber

01/19/2023, 11:51 PM

Going to do that shortly. I tested lowering the

Xmx

3G

from

6G

for both the gradle daemon and the compiler daemon and now I get this failure almost immediately:

Copy code

Failed to compile with Kotlin daemon: java.lang.RuntimeException: Could not connect to Kotlin compile daemon
	at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.compileWithDaemon(GradleKotlinCompilerWork.kt:192)
	at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.compileWithDaemonOrFallbackImpl(GradleKotlinCompilerWork.kt:150)
	at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.run(GradleKotlinCompilerWork.kt:123)
	at org.jetbrains.kotlin.compilerRunner.GradleCompilerRunnerWithWorkers$GradleKotlinCompilerWorkAction.execute(GradleCompilerRunnerWithWorkers.kt:74)
	at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:63)
	at org.gradle.workers.internal.NoIsolationWorkerFactory$1$1.create(NoIsolationWorkerFactory.java:66)
	at org.gradle.workers.internal.NoIsolationWorkerFactory$1$1.create(NoIsolationWorkerFactory.java:62)
	at org.gradle.internal.classloader.ClassLoaderUtils.executeInClassloader(ClassLoaderUtils.java:100)
	at org.gradle.workers.internal.NoIsolationWorkerFactory$1.lambda$execute$0(NoIsolationWorkerFactory.java:62)
	at org.gradle.workers.internal.AbstractWorker$1.call(AbstractWorker.java:44)
	at org.gradle.workers.internal.AbstractWorker$1.call(AbstractWorker.java:41)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
	at org.gradle.workers.internal.AbstractWorker.executeWrappedInBuildOperation(AbstractWorker.java:41)
	at org.gradle.workers.internal.NoIsolationWorkerFactory$1.execute(NoIsolationWorkerFactory.java:59)
	at org.gradle.workers.internal.DefaultWorkerExecutor.lambda$submitWork$0(DefaultWorkerExecutor.java:169)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.runExecution(DefaultConditionalExecutionQueue.java:187)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.access$700(DefaultConditionalExecutionQueue.java:120)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner$1.run(DefaultConditionalExecutionQueue.java:162)
	at org.gradle.internal.Factories$1.create(Factories.java:31)
	at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:249)
	at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
	at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:114)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.runBatch(DefaultConditionalExecutionQueue.java:157)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.run(DefaultConditionalExecutionQueue.java:126)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Using fallback strategy: Compile without Kotlin daemon
Try ./gradlew --stop if this issue persists.
warning: ATTENTION!


This mode is not recommended for production use,
as no stability/compatibility guarantees are given on
compiler or generated code. Use it at your own risk!

eygraber

01/20/2023, 12:09 AM

@tapchicoma it looks like there is no kotlin daemon log. It actually looks like the daemon never started at all.

eygraber

01/20/2023, 6:05 AM

After much debugging, there are two issues: 1. On GitHub Actions Ubuntu 20.04 runner, an Xmx of 6g for both Gradle and Kotlin is too high, because the runner has 7g of RAM. I solved this by lowering the Xmx, and now it works fine with the fallback strategy disabled (although still takes roughly the same amount of time) 2. On GitHub Actions Ubuntu 22.04 runner (which

ubuntu-latest

now points to), ~~no matter what I do with the Xmx it seems like the Kotlin daemon never runs~~ after a lot of fat trimming I was able to get this to work seemingly consistently but at the cost of increased build time. When it does fail there are no logs in

/tmp

and the OOM reaper didn't log anything about killing any process. Might be related to https://github.com/actions/runner-images/issues/6709

tapchicoma

01/20/2023, 9:12 AM

It actually looks like the daemon never started at all.

That is really strange. Could you get logs with Gradle

--debug

and share them?

eygraber

01/20/2023, 8:14 PM

I'm a little burnt out of Kotlin/Gradle right now...it's been a stressful few days of updating to 1.8/8.0 😅 I'll get to this as soon as I can, but likely won't be until mid next week

tapchicoma

01/21/2023, 10:52 AM

we have several cases reported with OOM in Metaspace. Probably you are affected as well

eygraber

01/22/2023, 12:19 AM

I was looking at jconsole with some local builds and didn't notice any issues with metaspace. I'll try again with some different scenarios. The GitHub Actions thing might have been transient, because I can't get it to happen again.

eygraber

01/22/2023, 3:27 AM

Guess I wasn't paying attention. There's something going on with metaspace. Is there a youtrack/github issue I can follow?

eygraber

01/22/2023, 4:23 AM

Left it running for longer; definitely something going on with metaspace. Also looks like the Kotlin daemon (but not the Gradle daemon) has some kind of leak with "CodeHeap non profiled nmethods" but I don't know too much about that (

ReservedCodeCacheSize

is set to

240m

)

tapchicoma

01/23/2023, 10:44 AM

You could track this issue: https://youtrack.jetbrains.com/issue/KT-56093

eygraber

01/23/2023, 6:39 PM

@tapchicoma would this cause the corresponding metaspace leak in the gradle daemon? I was running detekt, which I believe uses the Kotlin compiler in process.

tapchicoma

01/23/2023, 6:40 PM

yeah, most probably

2702 Views

Open in Slack

Previous Next