Hello folks! I am trying to upload a file to S3 via a pre-signed URL, using a PUT. I thought it coul...

# ktor

[Ioannis Mavroukakis]

Hello folks! I am trying to upload a file to S3 via a pre-signed URL, using a PUT. I thought it could be done via

setBody(File("some_file.large").readChannel())

unfortunately Ktor sets some headers that upset S3. So the only way out is to specify contentType and contentLength but I don't see how I can do this and keep the logic of readChannel. Any suggestions? Thanks!

[Ioannis Mavroukakis]

what I have come up with is

setBody(
                object : OutgoingContent.WriteChannelContent() {
                    override val contentType = OctetStream
                    override val contentLength = archive.fileSize()
                    override suspend fun writeTo(channel: ByteWriteChannel) {
                        val readChannel = archive.readChannel()
                        channel.use {
                            while (!readChannel.isClosedForRead) {
                                val packet = readChannel.readRemaining(1024)
                                while (!packet.isEmpty) {
                                    channel.writePacket(packet)
                                }
                            }
                        }
                    }
                },
            )

but I keep running out of memory while uploading a 1G file.

[Aleksei Tirman [JB]]

Could you please try to send a request body this way?

setBody(object : OutgoingContent.ReadChannelContent() {
    override val contentType: ContentType = ContentType.Application.OctetStream
    override val contentLength: Long = archive.fileSize()
    override fun readFrom(): ByteReadChannel = archive.readChannel()
})

[Ioannis Mavroukakis]

trying it now

[Ioannis Mavroukakis]

although I think I may have already given that a go..let's see 🙂

[Ioannis Mavroukakis]

btw I am on a memory limited device 😞 only have 2G to play with.

[Ioannis Mavroukakis]

nope, I ran out of memory

[Ioannis Mavroukakis]

I am using the Apache engine, maybe CIO will behave better..

[Ioannis Mavroukakis]

no joy I am running out of heap

[Ioannis Mavroukakis]

I'm uploading a 1G file, I am really hoping it's not trying to load all of it to memory 🙂 any suggestions @Aleksei Tirman [JB]?

[Aleksei Tirman [JB]]

Unfortunately, I cannot reproduce your problem. I’ve test it with the following code and the profiler showed that peak heap memory usage is about 350 MB:

val client = HttpClient(Apache) {
    install(HttpTimeout) {
        socketTimeoutMillis = 9999999
        requestTimeoutMillis = 9999999
    }
}

val archive = File("temp_1GB_file")
<http://client.post|client.post>("<https://httpbin.org/post>") {
    setBody(object : OutgoingContent.ReadChannelContent() {
        override val contentType: ContentType = ContentType.Application.OctetStream
        override val contentLength: Long = archive.length()
        override fun readFrom(): ByteReadChannel = archive.readChannel()
    })
}

[Aleksei Tirman [JB]]

Could you please file an issue with a sample project attached?

[Ioannis Mavroukakis]

@Aleksei Tirman [JB] thanks. Unfortunately that 350MB is nearly 1/5th of the available RAM on the device and there are other things running..

[Ioannis Mavroukakis]

I do exactly as you did, the only constraint is the RAM.

[Ioannis Mavroukakis]

AH, and running Java8 😭

[Aleksei Tirman [JB]]

I’ve created this issue to address this problem.

[Ioannis Mavroukakis]

you're a star. Apologies I couldn't provide something more representative, our project is commercial.

[Ioannis Mavroukakis]

I think the CIO engine suffers from the same issue btw.