https://kotlinlang.org logo
#coroutines
Title
# coroutines
l

louiscad

11/15/2019, 3:47 PM
Is there a way to read/write a file without blocking a thread at all that I could put in a simple extension function? Right now, the easiest way I know is blocking a thread from
<http://Dispatchers.IO|Dispatchers.IO>
and using
readText()
or
readBytes()
, but this still blocks a thread while we are waiting for storage and not the CPU.
d

diesieben07

11/15/2019, 3:51 PM
You could use Java's
AsynchronousFileChannel
, which is callback based and could be adapted to
suspend
functions.
👍 1
p

Paul Woitaschek

11/15/2019, 4:01 PM
I don't get it. Doesn't somewhere a thread has to be blocked while performing disc read?
o

octylFractal

11/15/2019, 4:04 PM
technically yes, but with async io (on the JVM) it is only ever one thread in total that waits for the IO, and then runs the callback. in theory it could be even better, because at the native level IO can be completely non blocking due to
select
, but in Java async io simply runs
select
forever in a jvm-managed thread
l

Luis Munoz

11/15/2019, 4:14 PM
Doesn't Vertx do this, which is like Nodejs on JVM where it has an evenloop and using epoll to just place a task on the queue of the eventloop when data is ready
o

octylFractal

11/15/2019, 5:47 PM
yes, but since java ... 8? there is native support for it using the aforementioned
AsynchronousFileChannel
s

StevieB

11/15/2019, 6:58 PM
There was this, I have a feeling it was deprecated/removed but might be a basis for something - https://github.com/Kotlin/kotlinx.coroutines/tree/87eaba8a287285d4c47f84c91df7671fcb58271f/integration/kotlinx-coroutines-nio
l

louiscad

11/15/2019, 8:48 PM
Interesting! @elizarov Is there a reason this Java NIO integration got removed since then?
e

elizarov

11/15/2019, 9:42 PM
AsynchronousFileChannel is not a good solution for async file IO.
And the APIs based on those channel were not easy to use
l

louiscad

11/15/2019, 11:59 PM
So you'd recommend me to keep using
<http://Dispatchers.IO|Dispatchers.IO> { file.readText() }
?
s

StevieB

11/16/2019, 12:45 AM
Yeah, if @elizarov says AsynchronousFileChannel is a bad solution then that's good enough for me but I would be very interested in knowing why that is.
e

elizarov

11/16/2019, 7:48 AM
First of all, it does not really work. On Linux there’s really no async file api that would work across different file systems. Essentially, what
AsynchronousFileChannel
does inside is not different from
<http://Dispatchers.IO|Dispatchers.IO> { file.readText() }
, but with much more overhead and much less convenenient API on top of that.
u

uli

11/16/2019, 2:21 PM
I think posix does not define non-blocking io on files. Only on sockets.
https://www.remlab.net/op/nonblock.shtml
Copy code
Regular files are always readable and they are also always writeable. This is clearly stated in the relevant POSIX specifications. I cannot stress this enough. Putting a regular file in non-blocking has ABSOLUTELY no effects other than changing one bit in the file flags.
Linux being the exception, its posix AIO implementation in glibc emulates async operations with user level threads, whereas its native async I/O interface (io_submit() etc.) are truly asynchronous all the way down to the driver, assuming the driver supports it.
l

louiscad

11/16/2019, 6:31 PM
So on Apple's platforms, we could use async filesystem I/O because its implementation being async, unlike Linux?
e

elizarov

11/16/2019, 7:04 PM
But you don’t write backend apps on Apple platforms anyway, so you should not really care
l

louiscad

11/17/2019, 1:56 PM
I wasn't thinking about writing a backend app but just making a program as efficient as possible so it can finally do more or allow other programs do more at the same time. But now, that makes me wondering… if it is more efficient to use macOS for I/O intensive apps rather than Linux, would that make macOS finally more cost efficient for backends? When Apple announced the new Mac Mini, they advertised rack/cluster capabilities and that some customers were doing it (e.g. for CI). They did the same when talking about the new Mac Pro.
e

elizarov

11/17/2019, 2:48 PM
It does not really matter. In fact, blocking IO is often more efficient (has higher throughput). The reason we want to use non-blocking IO for network is to scale our services to thousands of connections. It makes no difference for a typical disk IO usage.
🙏 1
g

gergo

11/18/2019, 9:03 AM
@elizarov May I have a question, what would you recommend if the file reading operation should be cancellable?
e

elizarov

11/18/2019, 9:08 AM
I’d recommend to structure your code if a way that you wouldn’t need a cancellable file IO. In a rare event when you need to read a really large file which takes a lot of time you can read it in limited-size blocks and call
ensureActive
between blocks to support cancellation.
👍 1
g

gergo

11/18/2019, 9:09 AM
I see, thank you!
e

elizarov

11/18/2019, 9:09 AM
Most applications don’t read large files and do not care whether file IO is cancellable or not.
l

louiscad

11/18/2019, 9:27 AM
If we were to read very big text files (like
View.java
from Android, ~30K lines of code), the best thing to do would be to use
bufferedReader()
, reading chunk by chunk and calling
ensureActive()
before reading each chunk?
e

elizarov

11/18/2019, 9:28 AM
I don’t think that 30K lines is big enough to really care. I’d say if it takes <1s don’t do anything special. No benefit.
l

louiscad

11/18/2019, 9:30 AM
A better example would be insanely huge log files, I recall waiting 5+ seconds to open one like these in SublimeText in the past, but yeah, I agree about your point that it's not really needed if it's sub-second.
e

elizarov

11/18/2019, 9:34 AM
Yes. If are writing log viewer app you’d better avoid reading it all into memory in the first place. Then if user does something like “search” you’d design it in such a way that it not just periodically checks for cancellation, but also periodically updates UI to report progress.
👍 1
5 Views