What's the best way to deal with rendering data lo...

# compose-desktop

[martmists]

What's the best way to deal with rendering data loaded off-thread? I'm trying to display the waveform of the current audio, with the following snippet:

val points by produceState(FloatArray(2) { 0f }) {
                        val format = AudioFormat(44100f, 16, 2, true, true)
                        val info = DataLine.Info(TargetDataLine::class.java, format)
                        val line = AudioSystem.getLine(info) as TargetDataLine
                        line.open(format)
                        line.start()

                        while (true) {
                            val array = ByteArray(1024)  // 256 samples/channel/frame
                            line.read(array, 0, array.size)

                            val buffer = ByteBuffer.wrap(array)
                            val floats = FloatArray(array.size / 4) { i ->
                                val l = buffer.getShort().toFloat()
                                val r = buffer.getShort().toFloat()
                                (l + r) * norm / Short.MAX_VALUE.toFloat()
                            }

                            value = floats
                            delay(1)  // delay required to not hang the main thread
                        }
                    }

However, this lags the application too much. On the other hand, I tried wrapping everything inside produceState in a launch(<http://Dispatchers.IO|Dispatchers.IO>) but that ended up only updating the scene at ~6fps.
As for displaying, I'm using this snippet:

Canvas(modifier = ...) {
                    for (i in 0 until points.lastIndex) {
                        val dx = i.toFloat() / points.size
                        val dy = points[i]
                        val x = dx * size.width
                        val y = size.height / 2 + dy * size.height / 2

                        val dx2 = (i + 1).toFloat() / points.size
                        val dy2 = points[i + 1]
                        val x2 = dx2 * size.width
                        val y2 = size.height / 2 + dy2 * size.height / 2

                        drawLine(
                            color = Color.Green,
                            start = Offset(x, y),
                            end = Offset(x2, y2),
                            strokeWidth = 10f
                        )
                    }
                }

What can I do to make this display at an acceptable framerate?

[Chrimaeon]

1. I’d not create a new byte array on each iteration - you set a new content anyway on the read.
2.

while (isActive) {
            value =
                withContext(Dispatchers.Default) {
                    line.read(array, 0, array.size)

                    val buffer = ByteBuffer.wrap(array)
                    FloatArray(array.size / 4) { i ->
                        val l = buffer.getShort().toFloat()
                        val r = buffer.getShort().toFloat()
                        (l + r) * norm / Short.MAX_VALUE.toFloat()
                    }
                }
        }

3. try to optimize buffer.getShort().toFloat() lots of boxing and unboxing happening here.
4. I’d make Short.MAX_VALUE.toFloat() a constant for the same reason.

[Chrimaeon]

So, I think your algorithm to extract the data needs to be optimized a lot to run at 16.6ms (time you have at 60 fps)

[Kirill Grouchnikov]

Also, split it to understand which part takes a lot of time, producing data or drawing on the canvas

[martmists]

looks like what takes time depends on the buffer size. On a small buffer it performs poorly because the buffer overruns, while large buffers take too long to draw with the Canvas (I also tried GenericShape+Surface), and there's no value in-between that satisfies both

[Michael Paus]

1. Do you really need updates at 60 Hz? Do you think your eyes can follow that?
2. Combine your data for one frame into a path which you draw with a single draw command. This will also look better than single line segments.

[martmists]

1. I'd at least prefer it if it had the same framerate as the rest of the application
2. Unfortunately Paths don't provide a way to get the width/height of their element, so that's not really an option

[Kirill Grouchnikov]

There’s the true framerate, and then there’s tricks employed in any sizable graphic interface to make it look like a much more performant thing that it is under the hood. Once you start bumping into the limits of the underlying hardware or software stack, there’s nothing wrong with cutting corners