https://kotlinlang.org logo
#io
Title
# io
d

Didier Villevalois

07/09/2022, 8:53 AM
Hi all! I observe a lot of fragmentation and duplication of effort around buffers. As there is no standard APIs for multiplatform buffers, you have to do a lot of work if you want to share a buffer across a stack of libraries. For instance, one of my use-cases is to: (1) receive a packet from a
ktor-network
socket, (2) parse a message using
google-flatbuffers
which contains a packed scalar array, (3) that is itself fed to
multik
to make a multi-dimensional array of it. Along all this processing line, it is possible to have a single byte array backing multiple buffers and not have to copy any bytes. However, it is a lot of work currently because there is no standard buffer API. For instance,
google-flatbuffers
is defining their own Buffer API to support this no-copy principle: https://github.com/google/flatbuffers/blob/master/kotlin/flatbuffers-kotlin/src/commonMain/kotlin/com/google/flatbuffers/kotlin/Buffers.kt In the above-mentioned use-case for instance, I have to implement my own
com.google.flatbuffers.kotlin.ReadBuffer
above `ktor-io`'s
<http://io.ktor.utils.io|io.ktor.utils.io>.core.ByteReadPacket
, etc Isn't it time that people from Okio, Ktor and others get together to define a standard Buffer API and revive
kotlinx-io
?? (Sorry for the kinda rant but it is a little irritating...)
a

altavir

07/09/2022, 10:57 AM
Indeed. But proper IO is really hard. So either community works more on this, or we wait for people free themselves from other responsibilities.
d

Didier Villevalois

07/09/2022, 11:25 AM
@altavir what would be needed to start a working group between the developers of Okio, Ktor, Binom IO, and others?
a

altavir

07/09/2022, 11:27 AM
I think that the first step is to start working on a design document, that would list required capabilities, a comparison of different implementations and the discussion of possible solutions. It is a huge work.
Currently one of the main blockers for common io is a compromise between synchronous and asynchronous operations. I wanted to work on this, but right now, I sadly do not have time for that.
d

Didier Villevalois

07/09/2022, 11:45 AM
I am really not an IO expert so I would not be able to work on this alone. Would you like to start drafting such a design document with me?
I though asynchronous IO was superior to synchronous IO. Shouldn't we focus on that, at least in the context of Kotlin having coroutines?
a

altavir

07/09/2022, 11:46 AM
Not at the moment. I've done some work with kotlinx-io. But the problem is that it is really compilated. Event drafting a technical note will take a lot of effort. For now, I think reporting issues here and in YouTrack.
Fully asynchronous buffer operations means huge performance pitfall in many cases.
d

Didier Villevalois

07/09/2022, 11:53 AM
If I understand correctly, ktor-io have a synchronous
Buffer
abstraction and an asynchronous
ByteChannel
abstraction. Is that the kind of compromise you spoke about?
a

altavir

07/09/2022, 11:56 AM
Not exactly. Buffer as itself is mostly useless. The question is in how does
Input
work. Does it block, while waiting for a new input or does it suspend. I believe that @e5l found kind of solution. But everyone is a bit occupied right now.
o

Oleg Yukhnevich

07/09/2022, 12:13 PM
On current moment there is an ongoing rework of ktor-io here: https://github.com/ktorio/ktor-io And we are trying to create a general purpose buffer API, which will both support working with platform specific API, high level API and also async api with zero copy Not sure, if it will be possible to make it new kotlinx-io, but who knows
👍 1
😍 1
d

Didier Villevalois

07/09/2022, 12:51 PM
@Oleg Yukhnevich this work will eventually replace parts of the current
ktor-io
in Ktor, right? Is there any way I can participate to this effort, or is it too early?
o

Oleg Yukhnevich

07/09/2022, 12:57 PM
Yeah, this is the idea of ktor-team Everything is open, if you have some ideas on what you want, I think you can create issues, based on current design Also sometimes there are prs there But, yes it's in early development still
👍 1
a

Aaron Todd

11/01/2022, 1:36 PM
Indeed
kotlinx-io
getting pulled from the roadmap was a major blow to working towards a common community solution for a KMP IO library. Would love to see
okio
and
ktor
folks (and community) work together as there is a lot of overlap.
3 Views