sometimes my app throws one of these stacktraces w...

# coroutines

[Dazai]

sometimes my app throws one of these stacktraces without any references to anywhere in my code, how would i even go about debugging this? thanks!

java.lang.NegativeArraySizeException: -1537060379
at <http://io.ktor.utils.io|io.ktor.utils.io>.core.StringsKt.readBytes(Strings.kt:165)
at <http://io.ktor.utils.io|io.ktor.utils.io>.core.StringsKt.readBytes$default(Strings.kt:162)
at io.ktor.client.call.SavedCallKt.save(SavedCall.kt:73)
at io.ktor.client.call.SavedCallKt$save$1.invokeSuspend(SavedCall.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:39)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

[Sam]

NegativeArraySizeException typically indicates an overflow, i.e. something tried to allocate an array which is bigger than Int.MAX_VALUE

[Sam]

If this is happening when making an HTTP call, it probably means that the response body is too big to fit in a byte array 😱

[Sam]

I'm not sure whether there's a way to get a better stacktrace and find out which specific call in your code is causing the problem

[Dazai]

in that case its probably definitely this function, but i don't see how it can create a byte array thats too big

suspend fun downloadFile(
    saveLocation: File,
    fileUrl: String,
    retries: Int = 2,
    delay: Long = Random().nextLong(2000, 5000),
    overwrite: Boolean = false
): Boolean {
    if (retries < 0) return false
    if (saveLocation.exists() && !overwrite) return true
    if (fileUrl == "null") return true

    saveLocation.delete()
    if (!saveLocation.createNewFile()) return false
    try {
        logger.info { "Attempting to download $fileUrl, $retries retries remaining" }
        val response: HttpResponse = client.get(fileUrl)
        val channel = response.bodyAsChannel()
        val byteBufferSize = 128000
        while (channel.availableForRead > 0 || !channel.isClosedForRead) {
            if (channel.availableForRead < byteBufferSize) {
                saveLocation.appendBytes(channel.readRemaining().readBytes())
            } else {
                saveLocation.appendBytes(channel.readPacket(byteBufferSize).readBytes())
            }
        }
        return true
    } catch (e: Exception) {
        e.printStackTrace()
        delay(delay)
        return downloadFile(saveLocation, fileUrl, retries - 1)
    }
}

[Dazai]

also any tips on breaking this apart? is the biggest function in my project

[Dazai]

so i'm not surprised that its probably the culprit

[Sam]

From the stack trace you shared, it looks like the problem is happening inside ktor, while downloading the response. The stack trace mentions SavedCall, which indicates to me that ktor is downloading the entire response into memory, rather than streaming it

[Sam]

In your example, you're using client.get

[Sam]

Be aware that that will download the entire response before returning. So even if it looks like you're streaming from a channel, you're actually just reading data that's already been download into memory.

[Sam]

Assuming this is ktor 2, you can use client.prepareGet(...).execute { ... } to stream the response instead (where the code to consume the response goes inside the execute block)

[Sam]

In theory, that will let you download things that are bigger than Int.MAX_VALUE bytes

[Sam]

Docs: https://ktor.io/docs/response.html#streaming

[christophsturm]

you can also stream in ktor1 by doing httpClient.get<HttpStatement>

[Dazai]

oh cool that makes sense, thanks for the help! I didn't realize it was loading it all into memory