So I noticed `Buffer` now uses a `ByteArray` inste...
# io
d
So I noticed
Buffer
now uses a
ByteArray
instead of
CPointer<ByteVar>
. Is
<http://kotlinx.io|kotlinx.io>
still going to provide an abstraction for off-heap memory or should I be looking into implementing my own?
a
Abstracting away off-heap buffers is a complicated thing. Also after discussions during the kotlinconf I came to conclusion that we need to separate internal buffers and user-side binary blocks. My proposal is to introduce a
Binary
class, which represents re-readable binary block and make any type of input be able to fill the Binary native to it. Meaning that if we are reading from byte stream, we are getting
ByteArray
and if we are reading from off-heap storage, we get off-heap binary.
d
Yeah, that's why I'm waiting for someone else to do it. 😁
a
I've actually done basic implementation in my fork. It still not good because I tried to use Buffer as a stable binary implementation, and it is not (it is discarded after being read). I will return to it in near future.
d
I might just do
expect
/
actual
for the
java.nio.Buffer
classes. The problem is I don't like the API of those classes very much.
a
Neither io development team. Still, a subset of nio Buffer api is OK. Not the question is about multiplatform and zero-cost file access
e
@Dominaezzz
Copy code
So I noticed Buffer now uses a ByteArray instead of CPointer<ByteVar> . Is <http://kotlinx.io|kotlinx.io> still going to provide an abstraction for off-heap memory or should I be looking into implementing my own?
We did it only in the native part, just because we want to use GC to simplify API in case of long term cache(and similar use cases). In other cases, we're still using preallocated memory pools
j
@Dominaezzz I'm really intertested in seeing what you come up with. NIO leaves big shoes to fill but also tries to cram c++ iterators into java language specifically, and has awful OO composition. The goal of NIO was some cross platform bridge to async and pointer-dependent blits alient to heap memory. Mapping epoll could be more direct than NIO, the wins were big overall but there are some headaches that can be researched by digging through the netty buffer history suchlike tricking NIO into nuking internal native maps and such. kernel handles really don't suffer from the same problems NIO creates.
d
So if I have something like,
glMapBuffer(size: ULong): CPointer<ByteVar>
. I can't use
<http://kotlinx.io|kotlinx.io>
, to represent this data. In the end that's all I really need.
e
Ok, I see. Could you tell me how you want to use it later? We should clarify the use case of such wrappers.
d
Essentially dereference it. • Get or set Byte, Float, Int, etc at an offset from the address. • Do some sort of memcpy between them. • Bulk read and write into primitive arrays, at offsets. All in native byte order. That's about it.
e
The 2 and 3 cases could be covered by using Input and Output directly, and we definitely should think about the first case.
It would be easier if we had some code samples of such cases to play around. If you have any examples or ideas of doing it we accepting any cases in
playground
module. Thanks for pointing!
d
I can setup an example but I'd need to pull in some dependencies. I'll try and make a PR.
e
Thanks again, and no problems with dependencies in
playground
module add this point