https://kotlinlang.org logo
Join Slack
Powered by
I have an OOM problem when decoding a very large J...
# ktor
r
I have an OOM problem when decoding a very large JSON response with Ktor client (Ktor3 on JVM). More in thread.
The exception:
Copy code
2025-01-27 10:34:50,641 [quartzScheduler_Worker-1] ERROR p.f.i.e.scheduler.ScheduledJob - io.ktor.serialization.JsonConvertException: Illegal input: Java heap space
io.ktor.serialization.JsonConvertException: Illegal input: Java heap space
        at io.ktor.serialization.kotlinx.KotlinxSerializationConverter.deserialize(KotlinxSerializationConverter.kt:77)
        at io.ktor.serialization.ContentConverterKt$deserialize$$inlined$map$1$2.emit(Emitters.kt:51)
        at kotlinx.coroutines.flow.FlowKt__BuildersKt$asFlow$$inlined$unsafeFlow$3.collect(SafeCollector.common.kt:111)
        at io.ktor.serialization.ContentConverterKt$deserialize$$inlined$map$1.collect(SafeCollector.common.kt:109)
        at kotlinx.coroutines.flow.FlowKt__ReduceKt.firstOrNull(Reduce.kt:247)
        at kotlinx.coroutines.flow.FlowKt.firstOrNull(Unknown Source)
        at io.ktor.serialization.ContentConverterKt.deserialize(ContentConverter.kt:99)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt.ContentNegotiation$lambda$13$convertResponse(ContentNegotiation.kt:234)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt.access$ContentNegotiation$lambda$13$convertResponse(ContentNegotiation.kt:1)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt$ContentNegotiation$2$2.invokeSuspend(ContentNegotiation.kt:249)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt$ContentNegotiation$2$2.invoke(ContentNegotiation.kt)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt$ContentNegotiation$2$2.invoke(ContentNegotiation.kt)
        at io.ktor.client.plugins.api.TransformResponseBodyHook$install$1.invokeSuspend(KtorCallContexts.kt:105)
        at io.ktor.client.plugins.api.TransformResponseBodyHook$install$1.invoke(KtorCallContexts.kt)
        at io.ktor.client.plugins.api.TransformResponseBodyHook$install$1.invoke(KtorCallContexts.kt)
        at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
        at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
        at io.ktor.client.HttpClient$4.invokeSuspend(HttpClient.kt:1379)
        at io.ktor.client.HttpClient$4.invoke(HttpClient.kt)
        at io.ktor.client.HttpClient$4.invoke(HttpClient.kt)
        at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
        at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
        at io.ktor.client.plugins.logging.ReceiveHook$Context.proceed(Logging.kt:290)
        at io.ktor.client.plugins.logging.LoggingKt$Logging$2$3.invokeSuspend(Logging.kt:209)
        at io.ktor.client.plugins.logging.LoggingKt$Logging$2$3.invoke(Logging.kt)
        at io.ktor.client.plugins.logging.LoggingKt$Logging$2$3.invoke(Logging.kt)
        at io.ktor.client.plugins.logging.ReceiveHook$install$1.invokeSuspend(Logging.kt:298)
        at io.ktor.client.plugins.logging.ReceiveHook$install$1.invoke(Logging.kt)
        at io.ktor.client.plugins.logging.ReceiveHook$install$1.invoke(Logging.kt)
        at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
        at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
        at io.ktor.client.plugins.ReceiveError$install$1.invokeSuspend(HttpCallValidator.kt:149)
        at io.ktor.client.plugins.ReceiveError$install$1.invoke(HttpCallValidator.kt)
        at io.ktor.client.plugins.ReceiveError$install$1.invoke(HttpCallValidator.kt)
        at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
        at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
        at io.ktor.util.pipeline.DebugPipelineContext.execute$ktor_utils(DebugPipelineContext.kt:63)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:79)
         at io.ktor.client.call.HttpClientCall.bodyNullable(HttpClientCall.kt:86)
        at pl.finn.integracja.edoreczenia.service.EdoreczeniaService.sprawdzDokumenty$lambda$7$pobierzMessage(EdoreczeniaService.kt:473)
        at pl.finn.integracja.edoreczenia.service.EdoreczeniaService.access$sprawdzDokumenty$lambda$7$pobierzMessage(EdoreczeniaService.kt:48)
        at pl.finn.integracja.edoreczenia.service.EdoreczeniaService$sprawdzDokumenty$2$pobierzMessage$1.invokeSuspend(EdoreczeniaService.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:47)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at pl.finn.integracja.edoreczenia.scheduler.ScheduledJob.executeInternal(ScheduledJob.kt:69)
        at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at kotlinx.io.internal._Utf8Kt.commonToUtf8String(-Utf8.kt:31)
        at kotlinx.io.internal._Utf8Kt.commonToUtf8String$default(-Utf8.kt:27)
        at kotlinx.io.Utf8Kt.commonReadUtf8(Utf8.kt:620)
        at kotlinx.io.Utf8Kt.readString(Utf8.kt:210)
        at io.ktor.utils.io.core.StringsKt.readText(Strings.kt:61)
        at io.ktor.utils.io.core.StringsKt.readText$default(Strings.kt:58)
        at io.ktor.serialization.kotlinx.KotlinxSerializationConverter.deserialize(KotlinxSerializationConverter.kt:69)
        at io.ktor.serialization.ContentConverterKt$deserialize$$inlined$map$1$2.emit(Emitters.kt:51)
        at kotlinx.coroutines.flow.FlowKt__BuildersKt$asFlow$$inlined$unsafeFlow$3.collect(SafeCollector.common.kt:111)
        at io.ktor.serialization.ContentConverterKt$deserialize$$inlined$map$1.collect(SafeCollector.common.kt:109)
        at kotlinx.coroutines.flow.FlowKt__ReduceKt.firstOrNull(Reduce.kt:247)
        at kotlinx.coroutines.flow.FlowKt.firstOrNull(Unknown Source)
        at io.ktor.serialization.ContentConverterKt.deserialize(ContentConverter.kt:99)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt.ContentNegotiation$lambda$13$convertResponse(ContentNegotiation.kt:234)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt.access$ContentNegotiation$lambda$13$convertResponse(ContentNegotiation.kt:1)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt$ContentNegotiation$2$2.invokeSuspend(ContentNegotiation.kt:249)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt$ContentNegotiation$2$2.invoke(ContentNegotiation.kt)
        at io.ktor.client.plugins.contentnegotiation.ContentNegotiationKt$ContentNegotiation$2$2.invoke(ContentNegotiation.kt)
        at io.ktor.client.plugins.api.TransformResponseBodyHook$install$1.invokeSuspend(KtorCallContexts.kt:105)
        at io.ktor.client.plugins.api.TransformResponseBodyHook$install$1.invoke(KtorCallContexts.kt)
        at io.ktor.client.plugins.api.TransformResponseBodyHook$install$1.invoke(KtorCallContexts.kt)
        at io.ktor.util.pipeline.DebugPipelineContext.proceedLoop(DebugPipelineContext.kt:79)
        at io.ktor.util.pipeline.DebugPipelineContext.proceed(DebugPipelineContext.kt:57)
The response in a JSON containing a few very large (~300MB) strings (base64 encoded files).
I can add 
-Xmx8G
jvm option to make it work. But that's a lot of RAM for my clients.
Is there a way to optimize memory usage?
d
Maybe you could try a json library based on streaming decoding like Moshi with Okio?
I think there's an open issue in KotlinX Serialization that they're working on streaming too, but I'm not sure about the status of it... it still might not explain why you need 8gb to run this, but maybe it might help... you might always be able to make your own decoder for Ktor, or use it's lower level streaming response receival.
b
You can deserialize the payload in this case without content negotiation, something like
Copy code
val input = response.body<InputStream>()
val payload = Json.decodeFromStream<Type>(input)
Also should ensure to skip saving the body:
Copy code
client.get("<http://myurl.com>") {
     skipSavingBody()
 }
thank you color 1
4 Views
Open in Slack
PreviousNext
Powered by