https://kotlinlang.org logo
#ktor
Title
# ktor
r

Rafał Kuźmiński

03/23/2021, 2:52 PM
I am trying to download big file using Ktor and streaming feature, but I encountered issue. Code:
Copy code
client.get<HttpStatement>(downloadUrl).execute {
                    var offset = 0
                    val byteBufferSize = 1024 * 100

                    val channel = it.receive<ByteReadChannel>()
                    val writeChannel = osmFile.writeChannel()
                    do {
                        val data = ByteArray(byteBufferSize)

                        val currentRead = channel.readAvailable(data, offset, byteBufferSize)

                        writeChannel.writeAvailable(data,offset,currentRead)


                        offset += currentRead

                        println("Pushed $offset bytes with current read $currentRead")
                    } while (currentRead >= 0)
                }
I see that it will download file only until byteBufferSize. I though it was limit for one package, but something in code is making this hard top limit. I though it would work like this: 1. set bufferSize limit for package 2. open write channel for file 3. create ByteArray to store bytes from one package 4. read from readChannel bytes, using current offset, and bufferSize limit, and write it to ByteArray 5. read from ByteArray and save those bytes into file Can someone explain me what I did wrong? Thanks! PS file is downloaded for 100 kb and it stops. If limit wouldn't be so perfect, it would crash with OutOfBounds exception
j

Julius

03/23/2021, 3:31 PM
r

Rafał Kuźmiński

03/23/2021, 4:14 PM
I think it is different issue, which was fixed with updated version of Ktor
I tried to download File using ByteBuffer as well, but it produces OutOfMemory exception, which is weird, because I'm allocating Buffer only once and data is written into file
Copy code
client.get<HttpStatement>(downloadUrl) {
                        timeout { requestTimeoutMillis = 2.minutes.toLongMilliseconds() }
                    }.execute {

                        val byteBufferSize = 1024 * 100
                        val channel = it.receive<ByteReadChannel>()
                        val writeChannel = osmFile.writeChannel()


                        var bytesRead = 0
                        val byteBuffer = ByteBuffer.allocate(byteBufferSize)

                        while (bytesRead != -1) {
                            bytesRead = channel.readAvailable(byteBuffer)
                            writeChannel.writeAvailable(byteBuffer)
                            byteBuffer.clear()
                        }
                        
                    }
Can someone from Ktor / Kotlinx io team explain why memory is getting filled?
8 Views