I m using ktor client with the CIO engine for my kt search c

Question

I m using ktor client with the CIO engine for my kt search client a rest client for opensearch and elasticsearch We recently switched to this with our main application Spring Boot and now running our multi threaded tests we run with ten or so junit threads I get a few weird exceptions that randomly fail small portion of the tests see stacktrace below After a bit of digging I narrowed the problem down to the CIO engine and I suspect it s not completely threadsafe somehow Switching to the Java engine makes the problem go away completely I haven t tried any of the other engines The previous version of the client would have used Elastic s RestHighLevel client which in turn uses Apache http client if I recall correctly So the main change here is that we are now using a different Rest client Looking at the documentation I wonder if there is any guidance on pros cons of each engine There seem to be a lot of them at least For reference this is the stack trace It looks to me like different threads are ending up trying to open the same port somehow or messing with the same connection ```java net BindException Can t assign requested address at java base sun nio ch Net connect0 Native Method at java base sun nio ch Net connect Net java 579 at java base sun nio ch Net connect Net java 586 at java base sun nio ch SocketChannelImpl connect SocketChannelImpl java 853 at io ktor network sockets SocketImpl connect$ktor network SocketImpl kt 44 at io ktor network sockets ConnectUtilsJvmKt connect ConnectUtilsJvm kt 21 at io ktor network sockets TcpSocketBuilder connect TcpSocketBuilder kt 37 at io ktor client engine cio ConnectionFactory connect ConnectionFactory kt 29 at io ktor client engine cio Endpoint$connect$2$connect$1 invokeSuspend Endpoint kt 156 at io ktor client engine cio Endpoint$connect$2$connect$1 invoke Endpoint kt at io ktor client engine cio Endpoint$connect$2$connect$1 invoke Endpoint kt at kotlinx coroutines intrinsics UndispatchedKt startUndispatchedOrReturnIgnoreTimeout Undispatched kt 100 at kotlinx coroutines TimeoutKt setupTimeout Timeout kt 146 at kotlinx coroutines TimeoutKt withTimeoutOrNull Timeout kt 103 at io ktor client engine cio Endpoint connect Endpoint kt 164 at io ktor client engine cio Endpoint makeDedicatedRequest Endpoint kt 98 at io ktor client engine cio Endpoint execute Endpoint kt 62 at io ktor client engine cio CIOEngine execute CIOEngine kt 84 at io ktor client engine HttpClientEngine$executeWithinCallContext$2 invokeSuspend HttpClientEngine kt 99 at Coroutine boundary at io ktor client engine HttpClientEngine$DefaultImpls executeWithinCallContext HttpClientEngine kt 100 at io ktor client engine HttpClientEngine$install$1 invokeSuspend HttpClientEngine kt 70 at io ktor client plugins HttpSend$DefaultSender execute HttpSend kt 138 at io ktor client plugins HttpRedirect$Plugin$install$1 invokeSuspend HttpRedirect kt 61 at io ktor client plugins HttpCallValidator$Companion$install$3 invokeSuspend HttpCallValidator kt 147 at io ktor client plugins HttpSend$Plugin$install$1 invokeSuspend HttpSend kt 104 at io ktor client plugins DefaultTransformKt$defaultTransformers$1 invokeSuspend DefaultTransform kt 53 at io ktor client plugins HttpCallValidator$Companion$install$1 invokeSuspend HttpCallValidator kt 126 at io ktor client plugins HttpRequestLifecycle$Plugin$install$1 invokeSuspend HttpRequestLifecycle kt 35 at io ktor client HttpClient execute$ktor client core HttpClient kt 191 at io ktor client statement HttpStatement executeUnsafe HttpStatement kt 108 at io ktor client statement HttpStatement execute HttpStatement kt 47 at com tryformation api graphql FormationClient executeRaw FormationClient kt 1037 at com tryformation api graphql FormationClient execute FormationClient kt 919 at com tryformation graphqlapi TestTeamAndClientCreator createTeam$suspendImpl APITest kt 211 at com tryformation graphqlapi analytics ArmyTestFixture create ArmyTestFixture kt 272 at com tryformation graphqlapi analytics ArmyTestFixture$armyWorkspace$2$1 invokeSuspend ArmyTestFixture kt 269 at com tryformation graphqlapi TestbeansKt$runTest$1 invokeSuspend testbeans kt 257 Caused by java net BindException Can t assign requested address at java base sun nio ch Net connect0 Native Method at java base sun nio ch Net connect Net java 579 at java base sun nio ch Net connect Net java 586 at java base sun nio ch SocketChannelImpl connect SocketChannelImpl java 853 at io ktor network sockets SocketImpl connect$ktor network SocketImpl kt 44 at io ktor network sockets ConnectUtilsJvmKt connect ConnectUtilsJvm kt 21 at io ktor network sockets TcpSocketBuilder connect TcpSocketBuilder kt 37 at io ktor client engine cio ConnectionFactory connect ConnectionFactory kt 29 at io ktor client engine cio Endpoint$connect$2$connect$1 invokeSuspend Endpoint kt 156 at io ktor client engine cio Endpoint$connect$2$connect$1 invoke Endpoint kt at io ktor client engine cio Endpoint$connect$2$connect$1 invoke Endpoint kt at kotlinx coroutines intrinsics UndispatchedKt startUndispatchedOrReturnIgnoreTimeout Undispatched kt 100 at kotlinx coroutines TimeoutKt setupTimeout Timeout kt 146 at kotlinx coroutines TimeoutKt withTimeoutOrNull Timeout kt 103 at io ktor client engine cio Endpoint connect Endpoint kt 164 at io ktor client engine cio Endpoint makeDedicatedRequest Endpoint kt 98 at io ktor client engine cio Endpoint execute Endpoint kt 62 at io ktor client engine cio CIOEngine execute CIOEngine kt 84 at io ktor client engine HttpClientEngine$executeWithinCallContext$2 invokeSuspend HttpClientEngine kt 99 at kotlin coroutines jvm internal BaseContinuationImpl resumeWith ContinuationImpl kt 33 at kotlinx coroutines DispatchedTask run DispatchedTask kt 106 at kotlinx coroutines internal LimitedDispatcher run LimitedDispatcher kt 42 at kotlinx coroutines scheduling TaskImpl run Tasks kt 95 at kotlinx coroutines scheduling CoroutineScheduler runSafely CoroutineScheduler kt 570 at kotlinx coroutines scheduling CoroutineScheduler$Worker executeTask CoroutineScheduler kt 750 at kotlinx coroutines scheduling CoroutineScheduler$Worker runWorker CoroutineScheduler kt 677 at kotlinx coroutines scheduling CoroutineScheduler$Worker run CoroutineScheduler kt 664 ```

Aleksei Tirman [JB] · Answer

Can you share a code snippet to reproduce the `BindException`

Jilles van Gurp · Answer

It s not a simple test It s hundreds of integration tests running with 10+ threads as fast as they can doing thousands of requests against Elasticsearch So not simple to reproduce

Jilles van Gurp · Answer

It s never the same tests that fail and they all pass when you run them individually

Aleksei Tirman [JB] · Answer

I ve reproduced it with the following code ```val client = HttpClient CIO coroutineScope 1 40000 map async client get <http localhost 8888> awaitAll ```

Aleksei Tirman [JB] · Answer

Here is the <https youtrack jetbrains com issue KTOR 5697|issue>

Jilles van Gurp · Answer

Thanks for this