I've got a `suspend fun` which produces `ByteArray` in 16 byte chunks. Reading them is kinda time-in...

# coroutines

[reactormonk]

I've got a suspend fun which produces ByteArray in 16 byte chunks. Reading them is kinda time-intensive, so I was wondering if there's a concept of a lazy channel of bytes, so I say "ready me X bytes" without it reading everything.

[Trevor Stone]

https://square.github.io/okio/ ?

[reactormonk]

Doesn't look like it plays well with suspend?

[Joffrey]

You could also use `ktor-io`'s ByteReadChannel

[Joffrey]

Or use a Flow<ByteArray> or similar. I'm not sure exactly what you're looking for

[Trevor Stone]

Yeah it might be helpful to add some more context to what you're trying to do

[reactormonk]

I have:

suspend fun read4Pages(location: Byte): ByteArray = ...

I'm currently using:

suspend fun readAll(from: Byte, to: Byte) = {
val reads = (to - from + 1) / 4
    return (1..(reads)).flatMap {
        val pageNr = it * 4
        read4Pages(pageNr).toList()
    }.toByteArray()
}

... which is quite slow, because it reads everything. Takes about ~ 3s. However, there's information in the pages on how far I have to read (currently it's about 20% of the total size), so I want to avoid reading everything.

[reactormonk]

However, there's also the possibility of having more than one record at a time, which basically consist of "type, length, data" triplets, and are stacked behind each other. So being able to treat it as one continuous block of bytes would be cool.

[reactormonk]

... and there's a record which says "no more records will follow", which allows me to abort reading, so I don't have to read everything.

[Trevor Stone]

Maybe I am being obtuse here, but this hasn't clarified much for me to be able to provide any meaningful advice. I'm not sure what you're reading from and if that matters, nor where the 3s number is coming from. If you're needing to read from disk or network and it is a huge amount of data then sometimes there is no getting around it.

It's unclear for read4Pages why it is returning a ByteArray which you then convert to a list, and then convert the group of lists back into a ByteArray.

[reactormonk]

I'm casting the collection types around, because the flatMap signature requires it.

[reactormonk]

I'm trying to improve that loop, ain't too nice anyway. I want to replace it with a stream, so I don't have to load everything, just as much as I require to get all the data. Most of it is zeroes.

[reactormonk]

The read4Pages is the base function which I wanna combine with something else to get a stream.

[uli]

Sounds like an unbuffered flow. The producer will only producer of the collector of ready to consume more.

[reactormonk]

Sounds about right, but I can't seem to find anything under that name in https://kotlinlang.org/api/kotlinx.coroutines/

[reactormonk]

Ah, https://stackoverflow.com/questions/65952256/kotlin-flow-buffer-capacity