my specific metric was to get function sizes down below escape analysis metrics and identify any potential concurrency wins for thread-based swim lanes. suspension appears to induce excessive context protection overheads which have large inline-challenging functions even on explicitly single-threaded code.

Yesterday we had a discussion with @Zelenyi about that and came to some decisions. The key difference of what I am doing to, say, python, is that tables are effectively immutable, so when you do any operation on columns, you are actually constructing a new table, which can reuse columns from the old table. I am not sure, that we actually have a valid case, when we change only some values in the table. I would welcome any examples.

cell values [in a dataframe] ^

Column mutation is possible at the moment, without violating immutability. You just create a new table, reusing unchanged columns. We can also add implementation specific mutation methods later.

I would welcome any suggestions about API.

this one https://github.com/jnorthrup/columnar

the input structuring is basic as is to be expected for a first cut. I don't know if this is anything more thana technology demo for the kotlin typealiasing capabilities tbh. i have soemthing that can scale up beyond where pandas dies, the premise is simplistic and portable to rust or c++ which makes sense as any significant expansion of scope with the lessons learned here.

i intended to test out y,x storage along with x,y as an orthogonal, to choose IO models appropriately, however kotlin and IO do not mix.

i eliminated capture and suspension overhead. it looks like the simplest possible parralellization to utilize underutilized cores is going to be java streams with threads. simpler still is port to c++ since it is now single threaded code, and insert openmp pragmas on a few inner loops.

i have considered that now the tabular y,x code is solid, i can create a file per column and unify these across combined cursors. one of my benchmarks converts 2.8 millionx7 cells into 1100x9800 give or take. so the intermediary would be generating, and mmap handles for at least 9800 additional small files. at this point those files no longer need the mmap code, and i don't have small-file access driver code at this time, so, until i see a really burning issue i'm ok with the existing costs up front. adding a seperate writable device is the cheapest IO upgrade

when i compare fwf textual input io to binary NIO field input of the same data, there is a single noinline keyword on the reducer eliminating any gains from hitting a smaller binary file, i believe.

this code is likely as optimal as jvm kotlin is going to get without an elaborate queuing of io zones for for batching up partial clusters