Hi All I like to know what could be the underlying issue for kotlinlang #ktor

Hi All, I like to know what could be the underlyin...

Victor Harlan Lacson

09/06/2023, 10:56 PM

Hi All, I like to know what could be the underlying issue for this scenario. Im having issue on an application run on docker image on kubernete, running it on my local is working fine. to give the context: I am triggering a long database transaction parsed from a csv file from a given source. I return http response 200 right away when the process is started. ( this process will not be executed again for a long time) processing the records around 1/10th of the file ktor throws an error:

java.util.concurrent.RejectedExecutionException: event executor terminated

at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)

at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351)

at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344)

at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)

at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)

at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)

at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:862)

at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:500)

at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)

at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)

at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:97)

at io.netty.channel.DefaultChannelPromise.setSuccess(DefaultChannelPromise.java:78)

at io.netty.channel.DefaultChannelPromise.setSuccess(DefaultChannelPromise.java:73)

at io.ktor.server.netty.NettyApplicationResponse.respondFromBytes$suspendImpl(NettyApplicationResponse.kt:61)

at io.ktor.server.netty.NettyApplicationResponse.respondFromBytes(NettyApplicationResponse.kt)

at io.ktor.server.netty.NettyApplicationResponse.respondNoContent$suspendImpl(NettyApplicationResponse.kt:73)

at io.ktor.server.netty.NettyApplicationResponse.respondNoContent(NettyApplicationResponse.kt)

at io.ktor.server.engine.BaseApplicationResponse.respondOutgoingContent$suspendImpl(BaseApplicationResponse.kt:151)

at io.ktor.server.engine.BaseApplicationResponse.respondOutgoingContent(BaseApplicationResponse.kt)

at io.ktor.server.netty.NettyApplicationResponse.respondOutgoingContent$suspendImpl(NettyApplicationResponse.kt:37)

at io.ktor.server.netty.NettyApplicationResponse.respondOutgoingContent(NettyApplicationResponse.kt)

at io.ktor.server.engine.BaseApplicationResponse$Companion$setupSendPipeline$1.invokeSuspend(BaseApplicationResponse.kt:317)

at io.ktor.server.engine.BaseApplicationResponse$Companion$setupSendPipeline$1.invoke(BaseApplicationResponse.kt)

at io.ktor.server.engine.BaseApplicationResponse$Companion$setupSendPipeline$1.invoke(BaseApplicationResponse.kt)

at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)

at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)

at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:88)

at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invokeSuspend(DefaultTransform.kt:29)

at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invoke(DefaultTransform.kt)

at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invoke(DefaultTransform.kt)

at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)

at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)

at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)

at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)

at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultInterceptors$1.invokeSuspend(BaseApplicationEngine.kt:148)

at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultInterceptors$1.invoke(BaseApplicationEngine.kt)

at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultInterceptors$1.invoke(BaseApplicationEngine.kt)

at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)

at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)

at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invokeSuspend(BaseApplicationEngine.kt:124)

at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)

at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)

at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)

at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)

at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)

at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)

at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)

at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)

at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)

at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)

at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invokeSuspend(DefaultEnginePipeline.kt:123)

at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)

at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)

at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)

at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)

at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)

at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)

at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)

at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)

at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)

at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)

at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:119)

at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)

at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)

at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:44)

at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:112)

at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:126)

at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)

at kotlinx.coroutines.BuildersKt.launch(Unknown Source)

at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)

at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)

at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)

at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:425)

at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

at io.ktor.server.netty.EventLoopGroupProxy$Companion.create$lambda$1$lambda$0(NettyApplicationEngine.kt:296)

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

at java.base/java.lang.Thread.run(Thread.java:833)

🧵 4

Aleksei Tirman [JB]

09/07/2023, 4:46 AM

Can you share the code of the handler where the background process is launched?

Victor Harlan Lacson

09/08/2023, 1:19 AM

Hi, I found the main issue here. so what actually happens here is all task was running on server main thread, and it seems incoming api calls are blocked, either pod was restarted due to api health cant be respond or the engine itself was forced to terminate the executor. My issue was related here, https://stackoverflow.com/questions/74772166/how-to-avoid-blocking-main-netty-thread-when-using-ktor-server so its always best (depends on scenario) to wrap your underlying process in another coroutine to make sure main thread is not blocked.

Open in Slack

Previous Next