Hey, I'm using in a service `http4k-connect-amazon...

# http4k

[DanielZ]

Hey,
I'm using in a service http4k-connect-amazon-s3 for copying files from one s3 bucket to another. Now I'm facing the situation to deal with some huge files (>5GB). Is it somehow possible to copy those file using http4k-connect-amazon-s3 or do I have to use aws sdk?
Background: I'm aware that CopyObject has a limitation of 5GB, so I tried GetObject and PutObject which fails OOM in StreamBody.
Any ideas? Do you have any experience with such cases?

[James Richardson]

Its not an http4k answer, but maybe rclone is more what you want?
If you want to use http4k, you'll need to use streaming apis, and also upload in multiple chunks to get decent performance probably.

[James Richardson]

Maybe there is a different architecture that could avoid having to copy large files around, as you'll also get charged for data in a lot of circumstances

[James Richardson]

Bear in mind that a lot of aws services have network that have fading performance - "up to x" types all use token bucket type thing to expire data transfer performance credits. Can be an issue when many big files.

[DanielZ]

Was afraid of that. I asked because for other stuff we are using http4k-connect APIs and I like the usability. Was assuming that we have to go a different path in that case.

[James Richardson]

Its not so much that it cannot do it, it can, but you'd have to implement all the functionality in terms of base aws calls. Other folks did that already quite well. Of course maybe there's a way that the architecture could be refactored so the bucket move isn't required.

[James Richardson]

rclone moveto s3:bucket/file s3:otherbucket/file

Would just work, assuming permissions appropriate

[James Richardson]

(Idk the deployment environment, could be hurdles!)

[DanielZ]

Thanks a lot. Looks we have to rework our architecture. Goal is that each service has it’s own storage, but we have to pass through some of the data.

[James Richardson]

Yeah.. each service should have its own state, but immutable data can definitely be stored, and references can be passed around instead. Depending on the size of the data file, sometimes its the compute that needs to be moved to the data, not the data to the compute.
The service could for example ask for a reference to the file and be given (after checking permissions) a time limited pre-signed url... then the next service.doesnt need to know anything about where and how the files are hosted..

[James Richardson]

Anyhow - good luck!

[DanielZ]

Thanks 😃 you triggered a lot of thoughts. I really appreciate it.

[James Richardson]

No worries! Glad to help