I am trying to download big file using Ktor and st...

# ktor

[Rafał Kuźmiński]

I am trying to download big file using Ktor and streaming feature, but I encountered issue.
Code:

client.get<HttpStatement>(downloadUrl).execute {
                    var offset = 0
                    val byteBufferSize = 1024 * 100

                    val channel = it.receive<ByteReadChannel>()
                    val writeChannel = osmFile.writeChannel()
                    do {
                        val data = ByteArray(byteBufferSize)

                        val currentRead = channel.readAvailable(data, offset, byteBufferSize)

                        writeChannel.writeAvailable(data,offset,currentRead)


                        offset += currentRead

                        println("Pushed $offset bytes with current read $currentRead")
                    } while (currentRead >= 0)
                }

I see that it will download file only until byteBufferSize. I though it was limit for one package, but something in code is making this hard top limit.

I though it would work like this:

1. set bufferSize limit for package
2. open write channel for file
3. create ByteArray to store bytes from one package
4. read from readChannel bytes, using current offset, and bufferSize limit, and write it to ByteArray
5. read from ByteArray and save those bytes into file
Can someone explain me what I did wrong? Thanks!

PS file is downloaded for 100 kb and it stops. If limit wouldn't be so perfect, it would crash with OutOfBounds exception

[Julius]

This link helped me:
https://github.com/ktorio/ktor/issues/1636

[Rafał Kuźmiński]

I think it is different issue, which was fixed with updated version of Ktor

[Rafał Kuźmiński]

I tried to download File using ByteBuffer as well, but it produces OutOfMemory exception, which is weird, because I'm allocating Buffer only once and data is written into file

client.get<HttpStatement>(downloadUrl) {
                        timeout { requestTimeoutMillis = 2.minutes.toLongMilliseconds() }
                    }.execute {

                        val byteBufferSize = 1024 * 100
                        val channel = it.receive<ByteReadChannel>()
                        val writeChannel = osmFile.writeChannel()


                        var bytesRead = 0
                        val byteBuffer = ByteBuffer.allocate(byteBufferSize)

                        while (bytesRead != -1) {
                            bytesRead = channel.readAvailable(byteBuffer)
                            writeChannel.writeAvailable(byteBuffer)
                            byteBuffer.clear()
                        }
                        
                    }

Can someone from Ktor / Kotlinx io team explain why memory is getting filled?