Hi all! I observe a lot of fragmentation and dupli...

# io

[Didier Villevalois]

Hi all! I observe a lot of fragmentation and duplication of effort around buffers.

As there is no standard APIs for multiplatform buffers, you have to do a lot of work if you want to share a buffer across a stack of libraries. For instance, one of my use-cases is to: (1) receive a packet from a ktor-network socket, (2) parse a message using google-flatbuffers which contains a packed scalar array, (3) that is itself fed to multik to make a multi-dimensional array of it. Along all this processing line, it is possible to have a single byte array backing multiple buffers and not have to copy any bytes.

However, it is a lot of work currently because there is no standard buffer API. For instance, google-flatbuffers is defining their own Buffer API to support this no-copy principle: https://github.com/google/flatbuffers/blob/master/kotlin/flatbuffers-kotlin/src/commonMain/kotlin/com/google/flatbuffers/kotlin/Buffers.kt

In the above-mentioned use-case for instance, I have to implement my own com.google.flatbuffers.kotlin.ReadBuffer above `ktor-io`'s <http://io.ktor.utils.io|io.ktor.utils.io>.core.ByteReadPacket, etc

Isn't it time that people from Okio, Ktor and others get together to define a standard Buffer API and revive kotlinx-io ??

(Sorry for the kinda rant but it is a little irritating...)

[altavir]

Indeed. But proper IO is really hard. So either community works more on this, or we wait for people free themselves from other responsibilities.

[Didier Villevalois]

@altavir what would be needed to start a working group between the developers of Okio, Ktor, Binom IO, and others?

[altavir]

I think that the first step is to start working on a design document, that would list required capabilities, a comparison of different implementations and the discussion of possible solutions. It is a huge work.

[altavir]

Currently one of the main blockers for common io is a compromise between synchronous and asynchronous operations. I wanted to work on this, but right now, I sadly do not have time for that.

[Didier Villevalois]

I am really not an IO expert so I would not be able to work on this alone. Would you like to start drafting such a design document with me?

[Didier Villevalois]

I though asynchronous IO was superior to synchronous IO. Shouldn't we focus on that, at least in the context of Kotlin having coroutines?

[altavir]

Not at the moment. I've done some work with kotlinx-io. But the problem is that it is really compilated. Event drafting a technical note will take a lot of effort. For now, I think reporting issues here and in YouTrack.

[altavir]

Fully asynchronous buffer operations means huge performance pitfall in many cases.

[Didier Villevalois]

If I understand correctly, ktor-io have a synchronous Buffer abstraction and an asynchronous ByteChannel abstraction. Is that the kind of compromise you spoke about?

[altavir]

Not exactly. Buffer as itself is mostly useless. The question is in how does Input work. Does it block, while waiting for a new input or does it suspend. I believe that @e5l found kind of solution. But everyone is a bit occupied right now.

[Oleg Yukhnevich]

On current moment there is an ongoing rework of ktor-io here: https://github.com/ktorio/ktor-io
And we are trying to create a general purpose buffer API, which will both support working with platform specific API, high level API and also async api with zero copy
Not sure, if it will be possible to make it new kotlinx-io, but who knows

[Didier Villevalois]

@Oleg Yukhnevich this work will eventually replace parts of the current ktor-io in Ktor, right? Is there any way I can participate to this effort, or is it too early?

[Oleg Yukhnevich]

Yeah, this is the idea of ktor-team
Everything is open, if you have some ideas on what you want, I think you can create issues, based on current design
Also sometimes there are prs there
But, yes it's in early development still

[Aaron Todd]

Indeed kotlinx-io getting pulled from the roadmap was a major blow to working towards a common community solution for a KMP IO library. Would love to see okio and ktor folks (and community) work together as there is a lot of overlap.