Hi all! I observe a lot of fragmentation and duplication of effort around buffers. As there is no standard APIs for multiplatform buffers, you have to do a lot of work if you want to share a buffer across a stack of libraries. For instance, one of my use-cases is to: (1) receive a packet from a

ktor-network

socket, (2) parse a message using

google-flatbuffers

which contains a packed scalar array, (3) that is itself fed to

multik

to make a multi-dimensional array of it. Along all this processing line, it is possible to have a single byte array backing multiple buffers and not have to copy any bytes. However, it is a lot of work currently because there is no standard buffer API. For instance,

google-flatbuffers

is defining their own Buffer API to support this no-copy principle: https://github.com/google/flatbuffers/blob/master/kotlin/flatbuffers-kotlin/src/commonMain/kotlin/com/google/flatbuffers/kotlin/Buffers.kt In the above-mentioned use-case for instance, I have to implement my own

com.google.flatbuffers.kotlin.ReadBuffer

above `ktor-io`'s

<http://io.ktor.utils.io|io.ktor.utils.io>.core.ByteReadPacket

, etc Isn't it time that people from Okio, Ktor and others get together to define a standard Buffer API and revive

kotlinx-io

?? (Sorry for the kinda rant but it is a little irritating...)

Indeed. But proper IO is really hard. So either community works more on this, or we wait for people free themselves from other responsibilities.

@altavir what would be needed to start a working group between the developers of Okio, Ktor, Binom IO, and others?

I think that the first step is to start working on a design document, that would list required capabilities, a comparison of different implementations and the discussion of possible solutions. It is a huge work.

I am really not an IO expert so I would not be able to work on this alone. Would you like to start drafting such a design document with me?

Not at the moment. I've done some work with kotlinx-io. But the problem is that it is really compilated. Event drafting a technical note will take a lot of effort. For now, I think reporting issues here and in YouTrack.

If I understand correctly, ktor-io have a synchronous

Buffer

abstraction and an asynchronous

ByteChannel

abstraction. Is that the kind of compromise you spoke about?

Not exactly. Buffer as itself is mostly useless. The question is in how does

Input

work. Does it block, while waiting for a new input or does it suspend. I believe that @e5l found kind of solution. But everyone is a bit occupied right now.

On current moment there is an ongoing rework of ktor-io here: https://github.com/ktorio/ktor-io And we are trying to create a general purpose buffer API, which will both support working with platform specific API, high level API and also async api with zero copy Not sure, if it will be possible to make it new kotlinx-io, but who knows

@Oleg Yukhnevich this work will eventually replace parts of the current

ktor-io

in Ktor, right? Is there any way I can participate to this effort, or is it too early?

Yeah, this is the idea of ktor-team Everything is open, if you have some ideas on what you want, I think you can create issues, based on current design Also sometimes there are prs there But, yes it's in early development still

Indeed

kotlinx-io

getting pulled from the roadmap was a major blow to working towards a common community solution for a KMP IO library. Would love to see

okio

and

ktor

folks (and community) work together as there is a lot of overlap.