https://kotlinlang.org logo
Title
k

Kris Wong

01/10/2020, 10:46 PM
oi... are you all aware that you can only use the ktor client from the main thread for native targets?
seems like a huge restriction
b

Brendan Weinstein

01/11/2020, 4:07 AM
There are a few bugs open on ktor for native-mt-1.3.3 I believe it will be fixed. If you swap in Ben's coroutineworker lib you can use ktor on a bg thread just fine
k

Kris Wong

01/11/2020, 3:23 PM
i am using that lib, actually
that's how I found the issue
Uncaught Kotlin exception: io.ktor.http.URLParserException: Fail to parse url: <redacted> Caused by: kotlin.native.IncorrectDereferenceException: Trying to access top level value not marked as @ThreadLocal or @SharedImmutable from non-main thread at 0 test.kexe 0x0000000101edb8e7 kfun:kotlin.Throwable.<init>(kotlin.String?)kotlin.Throwable + 87 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Throwable.kt:22:37) at 1 test.kexe 0x0000000101ed4ee5 kfun:kotlin.Exception.<init>(kotlin.String?)kotlin.Exception + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Exceptions.kt:23:44) at 2 test.kexe 0x0000000101ed4aa5 kfun:kotlin.RuntimeException.<init>(kotlin.String?)kotlin.RuntimeException + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Exceptions.kt:34:44) at 3 test.kexe 0x0000000101f07b55 kfun:kotlin.native.IncorrectDereferenceException.<init>(kotlin.String)kotlin.native.IncorrectDereferenceException + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/native/Runtime.kt:30:36) at 4 test.kexe 0x0000000101f25e99 ThrowIncorrectDereferenceException + 137 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/native/internal/RuntimeUtils.kt:87:11) at 5 test.kexe 0x0000000102200ed9 CheckIsMainThread + 25 at 6 test.kexe 0x00000001020e16a3 kfun:io.ktor.http.<get-URL_ALPHABET_CHARS>#internal + 35 (/opt/buildAgent/work/a85294440dc5c6e/ktor-http/common/src/io/ktor/http/Codecs.kt:11:9) at 7 test.kexe 0x00000001020e2ecd kfun:io.ktor.http.encodeURLPath@kotlin.String.()kotlin.String + 717 (/opt/buildAgent/work/a85294440dc5c6e/ktor-http/common/src/io/ktor/http/Codecs.kt:59:0) ...
the globals in Codecs.kt are not annotated properly
obviously an easy fix, though who knows where else there are similar issues
b

Brendan Weinstein

01/11/2020, 3:53 PM
Which version of coroutine library are you using?
k

Kris Wong

01/11/2020, 3:53 PM
1.3.3
b

Brendan Weinstein

01/11/2020, 3:56 PM
private suspend fun fetchRankingsStringFromNetwork() = CoroutineWorker.withContext(Dispatchers.Default) {
    try {
      val client = HttpClient {
        install(JsonFeature) {
          val kxs = KotlinxSerializer(Json.nonstrict)
          serializer = kxs
        }
      }
      val response = client.get<String> {
        url {
          takeFrom("<https://api.basebeta.com>")
          encodedPath = "/rankings"
        }
      }
      val settings = ServiceRegistry.appSettings.value!!
      settings.putString(KEY_RANKINGS_CACHE, response)
      response
    } catch (e: Exception) {
      kprint(e.message ?: "")
      null
    }
  }
Here is some real-world code (not the cleanest!), but maybe you can see something you are doing different?
ktor_version=1.3.0-rc2
kotlin_version=1.3.61
coroutines_version=1.3.3
k

Kris Wong

01/11/2020, 4:02 PM
yes, I am using
CoroutineWorker.execute
b

Brendan Weinstein

01/11/2020, 4:04 PM
Are you initializing the HttpClient there or somewhere else?
k

Kris Wong

01/11/2020, 4:06 PM
elsewhere
b

Brendan Weinstein

01/11/2020, 4:09 PM
I know it's not great, but I'd put the http initialization inside the same block and see if that resolves your issue. I believe it should, after confirming that, I'd work backwards to figure out how you want to make the client share-able across threads.
If you put a @ThreadLocal annotation on the httpClient then you'll end up with something similar, but will at least not instantiate the client more than once per thread. The docs say they may pull that annotation out in the future. If you store the client in at atomic reference then you'll have to freeze the client and it will give you a bunch of weird errors.
k

Kris Wong

01/11/2020, 4:11 PM
based on my understanding, the exception is indicative that it won't change the behavior by moving the client creation
I did put @ThreadLocal on the client creation
perhaps this was changed/fixed in 1.3, unsure
b

Brendan Weinstein

01/11/2020, 4:12 PM
"Caused by: kotlin.native.IncorrectDereferenceException: Trying to access top level value not marked as @ThreadLocal or @SharedImmutable from non-main thread" I've gotten this when trying to define an instance of a class at the top-level and then use it from a different thread. All top-level objects are by default defined on the main thread.
k

Kris Wong

01/11/2020, 4:13 PM
correct, which is why they need to be annotated
b

Brendan Weinstein

01/11/2020, 4:13 PM
So if you are defining the httpClient at the top-level, then you are defining it on the main thread. And if you access the client on a background thread, then k/n will freeze it since it is accessed by two threads.
k

Kris Wong

01/11/2020, 4:14 PM
changing the thread on which the client is created did not change the behavior
b

Brendan Weinstein

01/11/2020, 4:15 PM
Did you put the client in the same execute block?
global variables, unless specially marked, can be only accessed from the main thread (that is, the thread Kotlin/Native runtime was first initialized), if other thread access such a global, 
IncorrectDereferenceException
 is thrown
b

Brendan Weinstein

01/11/2020, 4:17 PM
I agree with your reading, @ThreadLocal should work here
https://github.com/Kotlin/kotlinx.coroutines/pull/1748/files -- there are some non-obvious things I've found in writing test code (eg switching coroutine context on the same thread will freeze the parent)
k

Kris Wong

01/11/2020, 4:18 PM
the problem values are constants so @SharedImmutable would solve it, but I don't know where all the globals may be
b

Brendan Weinstein

01/11/2020, 4:20 PM
My personal strategy has been assume that jetbrains will resolve these issues in the next release given the mountain of bugs filed, and keep httpclient initialization confined to the same execution block in the mean time. The ktor client code references a bunch of singletons itself and I suspect that plays a role in this exception and others.
k

Kris Wong

01/11/2020, 4:21 PM
yeah I am prepared to to use @ThreadLocal for the client itself
b

Brendan Weinstein

01/11/2020, 4:24 PM
ah you haven't tried that? I am curious if that works. I started off by testing 1.3.3-native-mt and it took long enough to get to the code above that my will to keep running different code configurations was depleted. But I believe I tried throwing @ThreadLocal on the client for 1.3.3-native-mt and ran into errors. Maybe it works with 1.3.3?
k

Kris Wong

01/11/2020, 4:24 PM
i did try that, yes
b

Brendan Weinstein

01/11/2020, 4:24 PM
ah k you are just saying you don't want to do dirty code to make this work. that's understandable too 🙂
k

Kris Wong

01/11/2020, 4:25 PM
i just backed the change out when it didn't change the result, so it's not annotated that way currently
n

napperley

01/12/2020, 12:37 AM
On some targets (eg Linux) the limitation isn't a big issue since Coroutines can be used.
With Kotlin JS for instance you are dealing with a single threaded environment (on the JS platform), which isn't a big issue since Coroutines can work in that environment.
Besides the use of the Worker concurrency model in Kotlin Native is optional. There are other ways that concurrency can be handled (eg Coroutines). Ktor is designed to work with Coroutines so the Ktor Client single threaded limitation isn't a big issue, as mentioned before.
k

Kris Wong

01/13/2020, 2:10 PM
agree to disagree on that point
🆗 1
i just tried upgrading to 1.3.0-rc2 and same issue
very interesting - @ThreadLocal combined with 1.3.0-rc2 seems to work
b

Brendan Weinstein

01/13/2020, 5:12 PM
Oh interesting, I'll give that a try later today. There was a cluster of comments on github issues last night for ktor -- it seems jetbrains folks have returned from holiday.
💼 1
k

Kris Wong

01/13/2020, 5:15 PM
definitely unexpected given that the error is rooted in how it's referencing globals