Can you upload files over krpc?

# kotlinx-rpc

[Dumitru Preguza]

Can you upload files over krpc?

[Alexander Sysoev]

Hi! No built-in way

[Dumitru Preguza]

Would make sense to create a streaming file upload over krpc(websickets)/grpc

[Alexander Sysoev]

Usually rpc is not a file streaming tool, rest is better for it as there are build-in resume api available. We'll take a not for it anyway though, thanks!

[Joost Klitsie]

yes you can 😄

[Joost Klitsie]

But not out of the box I guess. I implemented a simple image uploading method by using a flow of bytearrays ->
so in your rpc service:

suspend fun uploadFile(fileName: String, Flow<ByteArray>)

Then when you have your input file, a large bytearray, you chunk it up and put it into the flow. Then in the receiving end you collect that flow and put it back into a bytearray

[Joost Klitsie]

But I did run into issues that if I send the chunks too fast, it just hangs (I guess something bad happens) so you should add some delay between the chunks. So perhaps maybe it would be better to not do this if you want to do some serious work 🙂

[Alexander Sysoev]

Well, yeah, that's what I meant😅
This things are possible, yes

[Alexander Sysoev]

Would be interested to know what was wrong with "too fast"🤔

[Joost Klitsie]

So a pseudo snapshot of my code. Basically, if I send an image of a couple of MBs and I could log I was creating like 150 chunks, then perhaps after sending/receiving 70 chunks it simply didn't continue. Now I do not remember how I debugged this -> perhaps I simply logged how many chunks actually arrived on the server, which could imply either the client messed up or the server. What I know is that it got stuck. But when I added the delay as you see below, then it works fine. A small delay of 10 or something will also result in this stuck behavior. I don't think that I have debugged the actual network traffic, more like check what should go out and what finally came in.

bytes = byteArray.asFlow(CHUNK_SIZE)

fun ByteArray.asFlow(chunkSize: Int): Flow<ByteArray> {
    var count = 0
    return (0 until size step chunkSize).asFlow().map { startIndex ->
        count++
        val endIndex = minOf(startIndex + chunkSize, size)
        this.copyOfRange(startIndex, endIndex)
    }
}

const val CHUNK_SIZE = 64 * 1024 // 64KB

// Inside RpcClient
suspend fun uploadImage(
    authentication: SignedAuthentication,
    fileName: String,
    bytes: Flow<ByteArray>,
): PQResult<MediaRecord, CommonError>

rpcClient.uploadImage(
    authentication = authentication,
    fileName = fileName,
    bytes = bytes
        .onEach { delay(100) },
)