https://kotlinlang.org logo
Join Slack
Powered by
oi... are you all aware that you can only use the ...
# ktor
k
oi... are you all aware that you can only use the ktor client from the main thread for native targets?
seems like a huge restriction
b
There are a few bugs open on ktor for native-mt-1.3.3 I believe it will be fixed. If you swap in Ben's coroutineworker lib you can use ktor on a bg thread just fine
k
i am using that lib, actually
that's how I found the issue
Uncaught Kotlin exception: io.ktor.http.URLParserException: Fail to parse url: <redacted> Caused by: kotlin.native.IncorrectDereferenceException: Trying to access top level value not marked as @ThreadLocal or @SharedImmutable from non-main thread at 0 test.kexe 0x0000000101edb8e7 kfun:kotlin.Throwable.<init>(kotlin.String?)kotlin.Throwable + 87 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Throwable.kt2237) at 1 test.kexe 0x0000000101ed4ee5 kfun:kotlin.Exception.<init>(kotlin.String?)kotlin.Exception + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Exceptions.kt2344) at 2 test.kexe 0x0000000101ed4aa5 kfun:kotlin.RuntimeException.<init>(kotlin.String?)kotlin.RuntimeException + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Exceptions.kt3444) at 3 test.kexe 0x0000000101f07b55 kfun:kotlin.native.IncorrectDereferenceException.<init>(kotlin.String)kotlin.native.IncorrectDereferenceException + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/native/Runtime.kt3036) at 4 test.kexe 0x0000000101f25e99 ThrowIncorrectDereferenceException + 137 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/native/internal/RuntimeUtils.kt8711) at 5 test.kexe 0x0000000102200ed9 CheckIsMainThread + 25 at 6 test.kexe 0x00000001020e16a3 kfun:io.ktor.http.<get-URL_ALPHABET_CHARS>#internal + 35 (/opt/buildAgent/work/a85294440dc5c6e/ktor-http/common/src/io/ktor/http/Codecs.kt119) at 7 test.kexe 0x00000001020e2ecd kfun:io.ktor.http.encodeURLPath@kotlin.String.()kotlin.String + 717 (/opt/buildAgent/work/a85294440dc5c6e/ktor-http/common/src/io/ktor/http/Codecs.kt590) ...
the globals in Codecs.kt are not annotated properly
obviously an easy fix, though who knows where else there are similar issues
b
Which version of coroutine library are you using?
k
1.3.3
b
Copy code
private suspend fun fetchRankingsStringFromNetwork() = CoroutineWorker.withContext(Dispatchers.Default) {
    try {
      val client = HttpClient {
        install(JsonFeature) {
          val kxs = KotlinxSerializer(Json.nonstrict)
          serializer = kxs
        }
      }
      val response = client.get<String> {
        url {
          takeFrom("<https://api.basebeta.com>")
          encodedPath = "/rankings"
        }
      }
      val settings = ServiceRegistry.appSettings.value!!
      settings.putString(KEY_RANKINGS_CACHE, response)
      response
    } catch (e: Exception) {
      kprint(e.message ?: "")
      null
    }
  }
Here is some real-world code (not the cleanest!), but maybe you can see something you are doing different?
Copy code
ktor_version=1.3.0-rc2
kotlin_version=1.3.61
coroutines_version=1.3.3
k
yes, I am using 
CoroutineWorker.execute
b
Are you initializing the HttpClient there or somewhere else?
k
elsewhere
b
I know it's not great, but I'd put the http initialization inside the same block and see if that resolves your issue. I believe it should, after confirming that, I'd work backwards to figure out how you want to make the client share-able across threads.
If you put a @ThreadLocal annotation on the httpClient then you'll end up with something similar, but will at least not instantiate the client more than once per thread. The docs say they may pull that annotation out in the future. If you store the client in at atomic reference then you'll have to freeze the client and it will give you a bunch of weird errors.
k
based on my understanding, the exception is indicative that it won't change the behavior by moving the client creation
I did put @ThreadLocal on the client creation
perhaps this was changed/fixed in 1.3, unsure
b
"Caused by: kotlin.native.IncorrectDereferenceException: Trying to access top level value not marked as @ThreadLocal or @SharedImmutable from non-main thread" I've gotten this when trying to define an instance of a class at the top-level and then use it from a different thread. All top-level objects are by default defined on the main thread.
k
correct, which is why they need to be annotated
b
So if you are defining the httpClient at the top-level, then you are defining it on the main thread. And if you access the client on a background thread, then k/n will freeze it since it is accessed by two threads.
k
changing the thread on which the client is created did not change the behavior
b
Did you put the client in the same execute block?
k
global variables, unless specially marked, can be only accessed from the main thread (that is, the thread Kotlin/Native runtime was first initialized), if other thread access such a global, 
IncorrectDereferenceException
 is thrown
b
I agree with your reading, @ThreadLocal should work here
https://github.com/Kotlin/kotlinx.coroutines/pull/1748/files -- there are some non-obvious things I've found in writing test code (eg switching coroutine context on the same thread will freeze the parent)
k
the problem values are constants so @SharedImmutable would solve it, but I don't know where all the globals may be
b
My personal strategy has been assume that jetbrains will resolve these issues in the next release given the mountain of bugs filed, and keep httpclient initialization confined to the same execution block in the mean time. The ktor client code references a bunch of singletons itself and I suspect that plays a role in this exception and others.
k
yeah I am prepared to to use @ThreadLocal for the client itself
b
ah you haven't tried that? I am curious if that works. I started off by testing 1.3.3-native-mt and it took long enough to get to the code above that my will to keep running different code configurations was depleted. But I believe I tried throwing @ThreadLocal on the client for 1.3.3-native-mt and ran into errors. Maybe it works with 1.3.3?
k
i did try that, yes
b
ah k you are just saying you don't want to do dirty code to make this work. that's understandable too 🙂
k
i just backed the change out when it didn't change the result, so it's not annotated that way currently
n
On some targets (eg Linux) the limitation isn't a big issue since Coroutines can be used.
With Kotlin JS for instance you are dealing with a single threaded environment (on the JS platform), which isn't a big issue since Coroutines can work in that environment.
Besides the use of the Worker concurrency model in Kotlin Native is optional. There are other ways that concurrency can be handled (eg Coroutines). Ktor is designed to work with Coroutines so the Ktor Client single threaded limitation isn't a big issue, as mentioned before.
k
agree to disagree on that point
🆗 1
i just tried upgrading to 1.3.0-rc2 and same issue
very interesting - @ThreadLocal combined with 1.3.0-rc2 seems to work
b
Oh interesting, I'll give that a try later today. There was a cluster of comments on github issues last night for ktor -- it seems jetbrains folks have returned from holiday.
💼 1
k
definitely unexpected given that the error is rooted in how it's referencing globals
4 Views
Open in Slack
PreviousNext
Powered by