The docs: > The provisioning mechanism and all ...

# kotlin-toolchain

[jessewilson]

The docs:

The provisioning mechanism and all relevant behaviors in the Kotlin Toolchain are designed to be safe to use concurrently. This means you can run as many Kotlin CLI commands as you want in parallel, and they won't disturb each other.

I just crashed it by attempting to run two builds on the same project at the same time.

00:02.547 WARN  :api:resolveDependenciesWasmWasi Unexpected error occurred on the attempt to download kotlinx-coroutines-core-jvm-1.10.2.pom for dependency org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.10.2 (reason: java.io.IOException: Resource deadlock would occur): java.io.IOException: Resource deadlock would occur

[jessewilson]

Get a full stacktrace on the linked test failure. I think this is the relevant bit:

java.io.IOException: Resource deadlock would occur
          at java.base/sun.nio.ch.UnixFileDispatcherImpl.lock0(Native Method)
          at java.base/sun.nio.ch.UnixFileDispatcherImpl.lock(UnixFileDispatcherImpl.java:107)
          at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1693)
          at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1223)
          at org.jetbrains.amper.concurrency.DoubleLockKt$lockWithRetry$3.invokeSuspend$lambda$0(doubleLock.kt:236)
          at kotlinx.coroutines.InterruptibleKt.runInterruptibleInExpectedContext(Interruptible.kt:48)
          at kotlinx.coroutines.InterruptibleKt.access$runInterruptibleInExpectedContext(Interruptible.kt:1)
          at kotlinx.coroutines.InterruptibleKt$runInterruptible$2.invokeSuspend(Interruptible.kt:40)
          at kotlinx.coroutines.InterruptibleKt$runInterruptible$2.invoke(Interruptible.kt)
          at kotlinx.coroutines.InterruptibleKt$runInterruptible$2.invoke(Interruptible.kt)
          at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:66)
          at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)
          at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:157)
          at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
          at kotlinx.coroutines.InterruptibleKt.runInterruptible(Interruptible.kt:39)
          at kotlinx.coroutines.InterruptibleKt.runInterruptible$default(Interruptible.kt:36)
          at org.jetbrains.amper.concurrency.DoubleLockKt$lockWithRetry$3.invokeSuspend(doubleLock.kt:235)
          at org.jetbrains.amper.concurrency.DoubleLockKt$lockWithRetry$3.invoke(doubleLock.kt)
          at org.jetbrains.amper.concurrency.DoubleLockKt$lockWithRetry$3.invoke(doubleLock.kt)
          at org.jetbrains.amper.concurrency.DoubleLockKt.withRetry-WPwdCS8(doubleLock.kt:255)
          at org.jetbrains.amper.concurrency.DoubleLockKt.withRetry-WPwdCS8$default(doubleLock.kt:241)
          at org.jetbrains.amper.concurrency.DoubleLockKt.lockWithRetry(doubleLock.kt:232)
          at org.jetbrains.amper.concurrency.DoubleLockKt.getOrComputeWithDoubleLock(doubleLock.kt:331)
          at org.jetbrains.amper.concurrency.DoubleLockKt$getOrComputeWithDoubleLock$1.invokeSuspend(doubleLock.kt)

[joffrey]

@jessewilson thank you so much for this. One question, which OS are you running this on?

[joffrey]

The reason I'm asking is that there is a quirk with OS-level file locks on the JVM: the OS doesn't know about threads, and may fail because it thinks there could be a deadlock, but given the threading, there actually isn't. We cater for this specific case by looking for the Resource deadlock avoided message, and your error seems really close, but not quite the same. Hence my question.

[jessewilson]

Base is Linux AMD64 (x86_64)8 vCPU · 32 GB RAM
github.com/wasmcomputercompany/brevity/blob/…/Dockerfile

[jessewilson]

System software

Linux Ubuntu LTS

22.04

System

docker

29.3.0

docker-buildx

0.32.1

docker-compose

2.40.3

git-lfs

3.3.0

node

20

Pre-installed

aws-cli

2.13.0

[joffrey]

Awesome, thanks! I'll check this, maybe we need to check for this message as well. This is so brittle, I really don't like it, but I haven't found any other way. The JVM doesn't provide a special exception for this, so we have to check the message, and different OSes have different messages.

[joffrey]

By the way, I see it's a warning there. Did it fail your build, or was the dependency resolved eventually?

[joffrey]

For tracking: KTC-5765 IOException: Resource deadlock would occur

[jessewilson]

By the way, I see it's a warning there. Did it fail your build, or was the dependency resolved eventually?

The build failed. Concretely, the kotlin process returned a non-0 exit code. It printed a ton of warnings but nothing more severe than WARN, which seems like a different, complementary bug

[jessewilson]

If you don’t mind indulging me, what’s all happening here? I’m surprised that there’s file locking

[jessewilson]

Having never built a build system, I would have expected resource downloads to be implemented using an atomic-rename instead of a file lock

[joffrey]

I don't recall how the decision was made about file locks for dependency downloads, but I see a couple drawbacks with atomic moves. You might end up with duplicate downloads taking up space, just to give up on the downloaded data in the end. If you have big dependency trees, it might increase the required disk space unnecessarily. The atomic move after the download will also change the last-update timestamp, which might unnecessarily invalidate the cache of the concurrent processes (depending on the timing).

[joffrey]

If you're asking about the exception, the "Resource deadlock avoided" is an error coming from the OS itself when it thinks it detects deadlocks. The problem is that this check is at the level of processes and doesn't know about threads. So, if 2 processes both lock the same 2 files at the same time, but each in different threads, the system will think there is a deadlock even when there isn't:

• Process 1, thread A, acquires lock on file A.
• Process 2, thread B, acquires lock on file B.
• Process 1, thread B, tries to lock file B and blocks.
• Process 2, thread A, tries to lock file A and fails with the exception "Resource deadlock avoided".
Since we can't really prevent this, our best bet is just to retry a few times to see if the locks are eventually
released. If not, we can finally rethrow the exception.

[jessewilson]

Interesting! Thanks for the explanation. Yeah that sounds painful to mitigate.

[joffrey]

Yeah, having control flow based on exception messages gives me shivers 😅

[jessewilson]

Do the processes ever call each other? So tempting to over-engineer the heck out of this and have something other than the file system for coordination

[jessewilson]

Even that is probably extremely fragile? In practice everything is probably running inside layers of Docker abstractions and virtual file systems, where sockets and file systems are all weird

[joffrey]

If we're talking about Kotlin CLI processes, no they don't communicate for now. But also, the advantage of using the file locks for this is that, for some cases, we could benefit from the same cache in both the Kotlin Toolchain and KGP. For instance, the output of the commonizer is shared, and we lock on the same files to update the data. This could be generalized to more kotlin-related caches. Although, admittedly probably not the dependency resolution cache, given that all build tools have their own dependency cache.

[jessewilson]

One more, weirder, thing to consider. . . back in university I learned about deadlock prevention with the “dining philosophers” problem, and its solution - create a global order of locks and acquire them in order

[jessewilson]

For dependency downloads I wonder if that’s possible . . . you’re discovering new downloads as you go (transitive dependencies)

[jessewilson]

If you structured the program to always fetch graph structure first (.module and .pom files), with some secondary sort order within those things

[jessewilson]

Your program would say ‘I need to download the module file for coroutines and datetime and okio”, and acquire those locks in some consistent order

[jessewilson]

And when it discovers a 4th and 5th dependency, it’d need to release already-held locks before proceeding with the next round of module file downloads

[jessewilson]

Something to ponder, again if you wanna do something other than a giant retry loop

[joffrey]

Thanks for firing ideas! The resolution is already like this AFAIR: first resolving poms, then downloads. But poms are also downloads that need synchronization, and they are discovered as we go. Now, the locks are quite independent, it's on each dependency individually, and we don't keep hold of the lock on the parent while we resolve its descendants (again off the top of my head). So it's not like we're holding nested locks. It's just that the resolution is heavily parallel, so if we resolve, say coroutines and okio in parallel, and each take their own lock, and there happens to be another instance where both of these dependencies are also downloaded at the same time in different threads, the OS will scream deadlock even if everything is fine.

[joffrey]

So really, there is no deadlock at all, just the OS freaking out

[maartenh]

It sounds like both sides (KTC and OS) don't know enough about each other to properly work together on this. OS doesn't see the threads and KTC can't see a standardized reponse from the OS.
Frankly, while the OS may be freaking out unnecessarily, nobody would be happy with a policy that allowed the possibility of deadlocking.

I'm tempted to think that you could work around these issues by using something like a shared SQLite database to coordinate file locks. But that does add some overhead and I don't know if the performance slowdown would be a deal breaker.

[joffrey]

Frankly, while the OS may be freaking out unnecessarily, nobody would be happy with a policy that allowed the possibility of deadlocking

I disagree here. It's good to have some safety net that fails early when there are problems, but it's not acceptable that it fails when the problem is not present. I'd rather have a policy that allows some deadlocks if applications are poorly written, than a policy that crashes well-behaved applications unnecessarily.

[joffrey]

I'm tempted to think that you could work around these issues by using something like a shared SQLite database to coordinate file locks. But that does add some overhead and I don't know if the performance slowdown would be a deal breaker.

At the moment the only problem is that the error message for this OS error is unspecified, but with the 2 that we've seen, we should reasonably cover all supported OSes. These error messages don't change every day, and I'd be so bold as to say they are very unlikely to change at all. So, yes, it's in theory a brittle check, but in practice it's reasonable. So this check-message-and-retry behavior is pretty decent in the landscape of possibilities. Using more advanced ad-hoc synchronization mechanisms would complicate a lot the system and lose performance, just for a theoretical gain, so I'm not very keen to do that.
And yes, a slowdown for the dependency resolution is kind of a deal breaker, because it's a part of the build that's on the critical path.

[alexey.barsov]

FTR

We have changed the approach to file locking a bit and since now "Read deadlock avoided" and similar issues should not happen (the main idea was in replacing lock on the file with polling of tryLock)

[jessewilson]

Oh that's neat

[jessewilson]

The locking explanation in your commit is perfect
https://github.com/JetBrains/kotlin-toolchain/commit/e66d0c224617b83f5ef730d575f0310b986cfd40

[joffrey]

Kudos to Alexey for coming up with the idea!

[joffrey]

Note that we effectively avoid deadlock detection entirely here, because we never stop retrying. The outside contract of these method is to lock, so the caller has to be well behaved and make sure to avoid deadlocks. But since we don't acquire locks in a nested fashion, we should not have any problems in this regard. (Famous last words!)

[joffrey]

The nice thing is that, no matter how long dependency downloads are, we don't give up after an arbitrary time now. Until now, if your connection was slow enough to have a 10s download for a jar, you would be prone to this issue.