<@U5R239WUW> I’m thinking about porting snakeyaml-...

# serialization

[Piotr Krzemiński]

@charleskorn I’m thinking about porting snakeyaml-engine (the only JVM-only dependency of kaml) to Kotlin multiplatform, thanks to this kaml would become multiplatform (related request) and it would have a great impact on the Kotlin ecosystem. I don’t want to write a YAML parser/emitter from scratch because it’s too complex, but porting may actually work. I know it’s a serious endeavor, first porting (not sure if all relevant APIs are portable), then maintaining, handling security issues and so on. That’s why I’m curious if anyone ever considered doing it seriously - https://github.com/jurgc11/yakl seems to be dead - and if there’s anyone that would like to join me in this project. We could somehow split the work. So far I have only migrated to Gradle and automatically migrated several files to Kotlin, so you can say I’m just starting.

[Piotr Krzemiński]

these tasks come to my mind for the beginning:
• find transitive closure of Java classes in snake-yaml used by kaml - this way we can prioritize migrating these
• given the previous point, see how JVM-specific things can be replaced with KMP ones
• see how hard would it be to migrate existing snakeyaml-engine tests to KMP as well, so that we test for all targets

[Sebastian Schuberth]

Have you thought about joining forces with https://github.com/Him188/yamlkt instead?

[Piotr Krzemiński]

implementing a parser and a serializer from scratch? I’m afraid I cannot afford it time-wise. With snakeyaml-engine, a ton of problems is already solved, plus I’d like to pull fixes from the parent project

[Sebastian Schuberth]

I doubt that porting snakeyaml-engine is significantly less work than adding missing features to yamlkt. But of course that's just a guess, I haven't looked at either thoroughly.

[Piotr Krzemiński]

kaml wouldn’t be the only client of the ported version, it could be used in any KMP project

[Sebastian Schuberth]

Same for yamlkt.

[Piotr Krzemiński]

yamlkt forces requires using kotlinx.serialization right?

[Sebastian Schuberth]

Right. Although I'd say "requires" instead of "forces".

[Adam S]

what platforms do you want to target?

[Adam S]

how you could handle this is to refactor kaml so that all of the YAML code is in commonMain, and use actual/expect to define functions/classes that at the moment are provided by snakeyaml.

Once that’s done, and it’s all tested and works, try and find a suitable JS replacement library, and use that library to fulfil the same ‘expects’.

The benefit is that you don’t need to bother implementing or porting any code to Kotlin (which could be a large undertaking), you only need to provide a wrapper for snakeyaml/the JS lib using expect/actual functions.

[Piotr Krzemiński]

I'm thinking about actually porting snakeyaml-engine to platform-agnostic, common Kotlin, and this way support all platforms that Kotlin supports. I know it's much harder than what you propose with wrapping platform-specific libraries, but it also has its pros like consistent behavior across platforms, scalability with respect to platforms, and reusability outside of kotlinx.serialization. I'm gonna at least try this; maybe I'll hit a wall and I'll pass, but maybe I won't :)

[Adam S]

I quite enjoy converting Java to Kotlin, so I get where you’re coming from :)

do you have a repo? I’d be happy to help. We could convert a class at a time, and commit incrementally?

[Adam S]

it would be nice to convert a fully compliant library, of which there are only two… https://matrix.yaml.info/

[Piotr Krzemiński]

I do have the repo, but it's private for now because I'm still figuring out the strategy and trade-offs. For example, I'm wondering if we should:
• strive to keep the API perfectly compatible with the original library, or use Kotlin's features wherever possible. This includes such basic stuff as nullability, and others like data classes instead of POJOs
• migrate by hand or use the converter, or converter + minimal adjustments
• first migrate to Kotlin/JVM, and then make it platform-agnostic (2 sequential steps), or use platform-agnostic APIs during porting
I'm respectively for:
• Mimicking Java API as closely as possible, to make the Kotlin port a drop-in replacement, also be able to reuse snakeyaml docs and tests. At least for now, we can adjust this approach as we go
• Migrating using the converter, but if we see that it did something stupid or ugly, refactor manually. This is to save time and deliver something working faster, later on we can keep refactoring to make the code more idiomatic
• Migrating in two stages, first to Kotlin/JVM, which divides the porting project into smaller subproblems and will let us use existing JVM tests initially

Let me know what you think! 😀 I'd like to establish and consult such tenets beforehand, and make them public in the project README.

[Piotr Krzemiński]

Regarding the compliance, that's weird - snakeyaml advertises itself as a complete YAML 1.2 processor

[Piotr Krzemiński]

I'm open to choosing another library to port, but I don't have enough knowledge now to propose something else than snakeyaml-engine

[Adam S]

yeah, picking another library is absurdly theoretical and a lot more work. I was just poking around/being perfectionist.

If this project https://github.com/whyoleg/ffi-kotlin (or a similar one) takes off, then it would be worth it.

[Adam S]

with regards to migrating to Kotlin/JVM vs migrating to Kotlin Common - I would prefer setting up the structure of a Kotlin Multiplatform project, but only adding a single target: JVM. That way you can use Java libraries in commonMain. And then when the time comes to convert the library to another target, I can focus on replacing the Java functions with common replacements.

[Piotr Krzemiński]

Yes, makes sense to me! Here's the repo forked from snakeyaml-engine, with just several changes from me https://github.com/krzema12/snakeyaml-engine-kmp, feel free to contribute

[Piotr Krzemiński]

Also feel free to review the changes already merged, it's not a lot, and we could have a proper mutual code review process from now on

[Piotr Krzemiński]

@Adam S added you as a collaborator so you don't have to use a fork 😀

[Adam S]

8c8b22c9-542d-4155-964c-339acfa61534.mp4

8c8b22c9-542d-4155-964c-339acfa61534.mp4

[Adam S]

thanks! I’ll get cracking later on

[Adam S]

with respect to converting POJOs to data classes, and other such Kotlin improvements like nullability: I think it’s generally a good idea to make such changes because they interop really well with Java. But any major structural changes (anything that can’t be IDE auto-fixed, or find/replaced) are to be avoided. This is mostly so that the tests keep running and working, which is the most important thing.

[Adam S]

I converted a Json5 library from Java to Kotlin once, and it’s so satisfying to see the reduction in LOC

[Piotr Krzemiński]

Cool, good to know you have some experience with this!

[Piotr Krzemiński]

FYI, we’ve got a minimal example of the port working with Kotlin/JS. This test passes: https://github.com/krzema12/snakeyaml-engine-kmp/blob/daad9cae100a02d19a2aa37b4f89[…]rc/commonTest/kotlin/org/snakeyaml/engine/SimpleEndToEndTest.kt

kudos to @Adam S for great job, I almost just did the code review 😄

[Piotr Krzemiński]

you can say we have a PoC

[Piotr Krzemiński]

here's a little demo of using it in kaml- https://github.com/krzema12/kaml/blob/implement-JS-target/demo-consumer/src/main/kotlin/Main.kt, which prints:

====== Serialization ======
Serialized to YAML:
foo: "hey!"
bar: 123
baz:
  goo: 456.789
====== Deserialization ======
Deserialized from YAML:
SimpleClass(foo=deserialized-string, bar=987, baz=NestedClass(goo=11.22))

it has rough edges here and there (e.g. when reporting an issue through exceptions, there's a problem with instantiating the exception), but it looks promising!
CC @Adam S

[Adam S]

that’s amazing!

[Adam S]

RE: the exception, I think I’ve encountered & fixed that issue in https://github.com/krzema12/snakeyaml-engine-kmp/pull/59/files#diff-47256d64307ed2e4c53b6ccd0f934f8a6dd42c45349e32bafd0fd61ffc6acd66R21-R26. I’ve split it off into a new PR.

[Piotr Krzemiński]

https://github.com/krzema12/snakeyaml-engine-kmp/discussions/62 CC @charleskorn

[Piotr Krzemiński]

update: snakeyaml-engine-kmp has been published to Maven Central under it.krzeminski:snakeyaml-engine-kmp:2.7, and here’s a WIP PR that adds support for multiplatform to kaml, for now just for JS: https://github.com/krzema12/kaml/pull/1

[Piotr Krzemiński]

@charleskorn here's a PR to the kaml's original repo: Implement JS target - do you feel like taking it from here and leading it to a deployable state? You know kaml better 🙂 I think it would be good to just deploy it and mark that the JS target is extra-experimental (kaml itself is because it's still 0.x, and the JS target is experimental^2 because its dependency snakeyaml-engine-kmp is also experimental and WIP)

[Piotr Krzemiński]

@charleskorn ready to review and potentially merge as the first iteration: https://github.com/charleskorn/kaml/pull/437

[Piotr Krzemiński]

thanks for merging! works using the snapshot release, waiting for a release to use it in one of my projects please wait