oi are you all aware that you can only use the ktor client f

Question

oi are you all aware that you can only use the ktor client from the main thread for native targets

Kris Wong · Answer

seems like a huge restriction

Brendan Weinstein · Answer

There are a few bugs open on ktor for native mt 1 3 3 I believe it will be fixed If you swap in Ben s coroutineworker lib you can use ktor on a bg thread just fine

Kris Wong · Answer

i am using that lib actually

Kris Wong · Answer

that s how I found the issue

Kris Wong · Answer

Uncaught Kotlin exception io ktor http URLParserException Fail to parse url lt redacted gt Caused by kotlin native IncorrectDereferenceException Trying to access top level value not marked as ThreadLocal or SharedImmutable from non main thread at 0 test kexe 0x0000000101edb8e7 kfun kotlin Throwable lt init gt kotlin String kotlin Throwable + 87 Users teamcity3 buildAgent work 4d622a065c544371 runtime src main kotlin kotlin Throwable kt 22 37 at 1 test kexe 0x0000000101ed4ee5 kfun kotlin Exception lt init gt kotlin String kotlin Exception + 85 Users teamcity3 buildAgent work 4d622a065c544371 runtime src main kotlin kotlin Exceptions kt 23 44 at 2 test kexe 0x0000000101ed4aa5 kfun kotlin RuntimeException lt init gt kotlin String kotlin RuntimeException + 85 Users teamcity3 buildAgent work 4d622a065c544371 runtime src main kotlin kotlin Exceptions kt 34 44 at 3 test kexe 0x0000000101f07b55 kfun kotlin native IncorrectDereferenceException lt init gt kotlin String kotlin native IncorrectDereferenceException + 85 Users teamcity3 buildAgent work 4d622a065c544371 runtime src main kotlin kotlin native Runtime kt 30 36 at 4 test kexe 0x0000000101f25e99 ThrowIncorrectDereferenceException + 137 Users teamcity3 buildAgent work 4d622a065c544371 runtime src main kotlin kotlin native internal RuntimeUtils kt 87 11 at 5 test kexe 0x0000000102200ed9 CheckIsMainThread + 25 at 6 test kexe 0x00000001020e16a3 kfun io ktor http lt get URL ALPHABET CHARS gt internal + 35 opt buildAgent work a85294440dc5c6e ktor http common src io ktor http Codecs kt 11 9 at 7 test kexe 0x00000001020e2ecd kfun io ktor http encodeURLPath kotlin String kotlin String + 717 opt buildAgent work a85294440dc5c6e ktor http common src io ktor http Codecs kt 59 0

Kris Wong · Answer

the globals in Codecs kt are not annotated properly

Kris Wong · Answer

obviously an easy fix though who knows where else there are similar issues

Kris Wong · Answer

similar issue <https github com ktorio ktor issues 1544>

Kris Wong · Answer

Brendan Weinstein · Answer

Which version of coroutine library are you using

Kris Wong · Answer

1 3 3

Brendan Weinstein · Answer

``` private suspend fun fetchRankingsStringFromNetwork = CoroutineWorker withContext Dispatchers Default try val client = HttpClient install JsonFeature val kxs = KotlinxSerializer Json nonstrict serializer = kxs val response = client get lt String gt url takeFrom <https api basebeta com> encodedPath = rankings val settings = ServiceRegistry appSettings value settings putString KEY RANKINGS CACHE response response catch e Exception kprint e message null ```

Brendan Weinstein · Answer

Here is some real world code not the cleanest but maybe you can see something you are doing different

Brendan Weinstein · Answer

```ktor version=1 3 0 rc2 kotlin version=1 3 61 coroutines version=1 3 3```

Kris Wong · Answer

yes I am using `CoroutineWorker execute`

Brendan Weinstein · Answer

Are you initializing the HttpClient there or somewhere else

Kris Wong · Answer

elsewhere

Brendan Weinstein · Answer

I know it s not great but I d put the http initialization inside the same block and see if that resolves your issue I believe it should after confirming that I d work backwards to figure out how you want to make the client share able across threads

Brendan Weinstein · Answer

If you put a ThreadLocal annotation on the httpClient then you ll end up with something similar but will at least not instantiate the client more than once per thread The docs say they may pull that annotation out in the future If you store the client in at atomic reference then you ll have to freeze the client and it will give you a bunch of weird errors

Kris Wong · Answer

based on my understanding the exception is indicative that it won t change the behavior by moving the client creation

Kris Wong · Answer

I did put ThreadLocal on the client creation

Kris Wong · Answer

perhaps this was changed fixed in 1 3 unsure

Brendan Weinstein · Answer

Caused by kotlin native IncorrectDereferenceException Trying to access top level value not marked as ThreadLocal or SharedImmutable from non main thread I ve gotten this when trying to define an instance of a class at the top level and then use it from a different thread All top level objects are by default defined on the main thread

Kris Wong · Answer

correct which is why they need to be annotated

Brendan Weinstein · Answer

So if you are defining the httpClient at the top level then you are defining it on the main thread And if you access the client on a background thread then k n will freeze it since it is accessed by two threads

Kris Wong · Answer

changing the thread on which the client is created did not change the behavior

Brendan Weinstein · Answer

Did you put the client in the same execute block

Kris Wong · Answer

Kris Wong · Answer

global variables unless specially marked can be only accessed from the main thread that is the thread Kotlin Native runtime was first initialized if other thread access such a global `IncorrectDereferenceException` is thrown

Brendan Weinstein · Answer

I agree with your reading ThreadLocal should work here

Brendan Weinstein · Answer

<https github com Kotlin kotlinx coroutines pull 1748 files> there are some non obvious things I ve found in writing test code eg switching coroutine context on the same thread will freeze the parent

Kris Wong · Answer

the problem values are constants so SharedImmutable would solve it but I don t know where all the globals may be

Brendan Weinstein · Answer

My personal strategy has been assume that jetbrains will resolve these issues in the next release given the mountain of bugs filed and keep httpclient initialization confined to the same execution block in the mean time The ktor client code references a bunch of singletons itself and I suspect that plays a role in this exception and others

Kris Wong · Answer

yeah I am prepared to to use ThreadLocal for the client itself

Brendan Weinstein · Answer

ah you haven t tried that I am curious if that works I started off by testing 1 3 3 native mt and it took long enough to get to the code above that my will to keep running different code configurations was depleted But I believe I tried throwing ThreadLocal on the client for 1 3 3 native mt and ran into errors Maybe it works with 1 3 3

Kris Wong · Answer

i did try that yes

Brendan Weinstein · Answer

ah k you are just saying you don t want to do dirty code to make this work that s understandable too slightly smiling face

Kris Wong · Answer

i just backed the change out when it didn t change the result so it s not annotated that way currently

napperley · Answer

On some targets eg Linux the limitation isn t a big issue since Coroutines can be used

napperley · Answer

With Kotlin JS for instance you are dealing with a single threaded environment on the JS platform which isn t a big issue since Coroutines can work in that environment

napperley · Answer

Besides the use of the Worker concurrency model in Kotlin Native is optional There are other ways that concurrency can be handled eg Coroutines Ktor is designed to work with Coroutines so the Ktor Client single threaded limitation isn t a big issue as mentioned before

Kris Wong · Answer

agree to disagree on that point

Kris Wong · Answer

i just tried upgrading to 1 3 0 rc2 and same issue

Kris Wong · Answer

very interesting ThreadLocal combined with 1 3 0 rc2 seems to work

Brendan Weinstein · Answer

Oh interesting I ll give that a try later today There was a cluster of comments on github issues last night for ktor it seems jetbrains folks have returned from holiday

Kris Wong · Answer

definitely unexpected given that the error is rooted in how it s referencing globals