i prefer bash to python personally, but kotlin is not as generally handy or comfortable for simple glue code.

in fact, given the jdk's serialization headaches and the laggin native progress, it's really only for building shiny trophy examples of how to better express java than java.

There are no problem with serializarion in JDK and especially not in kotlin. We can discuss it in a separate thread.

explicit protobuf tooling is also kind of a sacrifice of enjoyable kotlin. im not really up to date on the specifics of kotlin serdes+tmfm but it seems pretty exciting. I think fixing hazelcast for kotlin is a great target for hazelcast inc. to keep current. the crossroads described by the 3-4 different models hazelcast belongs in more begs for serialization and coroutines to become externalized and distributed first class network smalltalk object descriptions, or whatever the hell works (toward first class kotlin)

the usecases of running callable<Pair<String,myKotlinDataClass>> is ussuallly enough to cause kotlin rewrites to "working" serialization. the deficit is in hazelcast serializable java using java as a contract for remote execution and object description, and kotlin being only partially inside the box

in https://github.com/0xCopy/prautobeans i have done the protobuff academia to allow protobuf definitions to shape snugly fit binary serialization packets as javabeans that don't choke GWT (a concept called autobeans)

rather than leverage protobuffers i stuck with off-heap capable constructs in NIO, protobuff uses heap objects.

in jvm-land you can define object proxies cheaply and that's really all you need to satisfy the proto IDL

it's been a few more than a few more years since i had the energy to change the world on this scale.

i honestly prefer hazelcast's Serialization interface solution, except that kotlin colors outside the lines and hazelcast forces you to write non-benefitical kotlin wrappers around distributed java objects. i had forgotten how much work is involved in maven generators with proto until now

a kotlin class like this one arises from something to take the hand-off from jdbc, generify it, and then slowly descend the spiral of despair with hazelcast enablement incrementally introducing such things as "turds" to the object model to be transient or explicit serialization workarounds

rather than "just" settling for kotlin with a hack like this, I'd love to mercilessly refactor in Arrow-kt expressions as well, where time permits. my dayjob affords me a few minutes at a time to add commits to this open source where im specifically not violating NDA's, and this is pre-existing of this engagement, again, a stable work from a long time ago getting new use.

having a hazelcast serialization strategy that for instance kills destructuring syntax or forces "turd" get() methods is only enjoyable once or twice, but remains a code-smell

a while ago i gave a look at borrowing the jvm compiler and building a compositional ISAM struct or otherwise, a wire format from declaratives. https://github.com/jnorthrup/1xam/blob/master/README.md#1xam i got as far as applying this generator to the subl-lisp object hierarchy of the CYC expert system in a related project on github (bitecode). the focus of this, again, was off-heap constructs first, and bounded chunks from which to assemble bigger assemblies for what would be called streaming architectures today.

I am yet not sure how your

wire format from declaratives

fits in, and too short on time now, may check later.

@jimn does Apache Ignite suffer the same issues you see w/ Hazelcast?

Having box format binary serialization is not new, and encourages performance, but for whatever reason academia and predicate calculus rely exclusively on string parsers

i only put this codebase down in august, but the details were vague until i checked back to look at the code again today. so the codebase i have is a utility to convert formats, sometimes to couchdb, sometimes to pandas formats, generally from a direct sql source. the more recent additions have been all-kotlin, which is not always as simple as ctl-alt-K, so the original simplest-possible java code snippets are still relevant in ways that were awkward for porting to kotlin directly. most recently, i have been moving hundreds of gigs of relational data to .. couchdb as a parking spot to come back to later. this was convenient for unrelated projects not once but three times ina row, temporary or expensive access to a database with too much legacy and only a few meaningful data sets sometimes among thousands of tables; so the capability of doing complete copies and subsequent delta modifications from sql to couchdb for an operational migration earlier this year as ReiterateDb https://github.com/jnorthrup/jdbc2json/blob/19a2119d7c6eeedf795c3195924a35370216ee66/src/main/java/com/fnreport/ReiterateDb.kt#L22 and more recently two research/analytics jobs on monster legacy data stores which warranted some expansion of SqlExecToJson to JdbcToCouchDbBulk https://github.com/jnorthrup/jdbc2json/blob/844f9af8e1c50aa7fc42a3f216b0fb6560a155f2/src/main/java/com/fnreport/JdbcToCouchDbBulk.kt#L16 built to handle sick primary keys and store data-only rows with column-labelled couchdb views. I'm not really biased toward couchdb all told, but i use pouchdb syncronoization for some production web code and couchdb uses as little as curl as a database driver, which is code-agnostic. so things were great, everything was falling into place, and now i have half terabyte of raltional data in couchdb and projects are spinning up rapidly here; time for one more metadata addressable catalog of my ongoing soup to move the jdbc metadata to hazelcast and start to scavenge vm's around the company where i can squirrel some memory grid state while i figure out how to make this useful for others. This usecases is where kotlin serialization is a hard show-stopper - instead of running the metadata extraction and throwing it in hazelcast, i think i just checked in the jdbc url to a hazelcast Map and called it quits for now. wthere are a few kotlin hacks like QueryToFlat and QueryToFeather which interface directly ahead of python, pandas, and tensorflow here. generally speaking, pandas is a glorified groupby tool for numpy arrays. if you can take a step back and leverage sql for group-by you may be able to bypass python entirely and make a quick line to google bigquery and get the simplest possible linear regression before going full-keras

the packaging methodology of building a uber-jar was feasable in the java 6 days, with like 8 years between vm changes - but shade and assemblies and maven have become brittle so the code creates a lib/*.jar structure and the bash scripts use a wildcard classpath; this way no collisions of manifests or multiple versions. having a repl, and having a distributed mesh of dataset metadata, and having a reasonably concise grammar for REPL interrogation by shell script seems like the next reasonable pragmatic step. it may not be as sexy as a whole math library, but i don't have coworkers who are technical and i need to make repeatable scripts with steps that can be repeated from an email alone

this is far from complete and i have no real toy examples to share, but the datasets of 100-200 megs sometimes take an hour or more for regular python groupby fixtures because of RAM and swapping. Upgrading from 32 gigs to 128 on the server may improve some of these datasets but we're starting to see features numbering in the 10's of thousands and even 128GB is not enough RAM for pandas to synthesize in RAM alone

divide and conquer with hazelcast would not be unthinkable once the jdbc sources and couchdb extractions are floating around in n=3 copies of network grid storage between idle machines here

i don't have an end-to-end jvm based keras adaptation that i can simply port my stuff to kotlin; and im not the guy who invented deep learning so im still coming up to speed on the problem solving methodology steps, most notably it seems like sample code is a ticking time-bomb because the problems lack representative hardware inconveniences of useful datasets

@Amir Gur @altavir basically Kotlin is missing a no-bullshit dht to approximate a p2p kvstore on top of plain old map interface. Hazel cast success here for jvm, but forces Kotlin to leave all of its features at the door and use a Java interface. Fixing Kotlin serialization may be less useful than just writing a portable Kotlin p2p bus for serialized objects.