Here is our KMP incremental Markdown parser (and m...

# feed

[Kazik Pogoda]

Here is our KMP incremental Markdown parser (and much, much more - HTML / XML / XSLT-like DSL). Handy for streaming LLM inference output:

https://github.com/xemantic/markanywhere

Few things still need to be fixed - e.g MathML support, but it is already fully implementing all the GitHub Flavoured Markdown Spec (with marked divergence required for real time streaming without buffering). Can be verified here:

https://xemantic.github.io/xemantic-markdown-editor/

[Anonymike]

Looks like a great start. We currently use the Jetbrains markdown library and have a layer of custom markdown handling, mainly for different markdown flavour extensions and things like generically adding classes or other common properties for providing information to a renderer of some sort. I'll be checking out what it would take to add more events or parsing additional metadata. to add to the event model.

Thanks for sharing the work!

[Anonymike]

I've also always been a fan of StaX / pull parsers as well so love that mindset haha

[Kazik Pogoda]

My assumption here is that LLMs would predominantly avoid any non-incremental Markdown when streaming the output (e.g. setext headers). If not, it can be still enforced in the system prompt. The rest of typical Markdown output would be GFM compliant, and almost fully streamable in real time (except for links, which still need buffering). In addition I want markanywhere to handle scenarios like 0-tools architecture (see undocumented Anthropic Markdown Language). Which then can be interpreted as Kotlin Script in our harnesses.

https://github.com/xemantic/golem-xiv

[altavir]

I've been searching for a lightweight multiplatform markdown library for some time. JB library is sadly poorly maintained and modularity there is broken (for example one can't use custom HTML renderer). Do you plan to support non-standard Markdown features like tables and custom tiple quote blocks?

[Kazik Pogoda]

@altavir What kind of tables do you have in mind?

@Test
    fun `should parse table`() = runTest {
        // given

        val textFlow = """
            | Column 1 | Column 2 |
            |----------|----------|
            | Cell 1   | Cell 2   |
            | Cell 3   | Cell 4   |
        """.trimIndent().chunkedRandomly().asFlow()

        // when
        val parsed = textFlow.parse()

        // then
        parsed.mergeAdjacentText() sameAs semanticEvents {
            "table" {
                "thead" {
                    "tr" {
                        "th" { +"Column 1" }
                        "th" { +"Column 2" }
                    }
                }
                "tbody" {
                    "tr" {
                        "td" { +"Cell 1" }
                        "td" { +"Cell 2" }
                    }
                    "tr" {
                        "td" { +"Cell 3" }
                        "td" { +"Cell 4" }
                    }
                }
            }
        }
    }

https://github.com/xemantic/markanywhere/blob/37106520a31aaded08ccb5dad587fe3b83645787/markanywhere-parse/src/commonTest/kotlin/MarkanywhereParserTest.kt#L960

Something else than this?

[Kazik Pogoda]

Here is the comprehensive answer from the agent:

https://github.com/xemantic/markanywhere/issues/41

[Kazik Pogoda]

And you can quickly check what works here:

https://xemantic.github.io/xemantic-markdown-editor/

[altavir]

Tables are just an example. They are not a part of CommonMark, but an extension. The question is if there is mechanism to support extensions in your library or you hardcode one of MarkDown variants?

[altavir]

Fore example, your sample site shows some latex samples, but they do no work. $$ notation is not always good, so in most modular markdown parsers, they are optional, so they could be switched on or of. Is it possible in your case?

[Kazik Pogoda]

My approach is more pragmatic at this point. The library is a markdown parser by chance. Primarily it delivers a model of semantic event streams and tooling for processing of these streams. Streaming LLM output as it happens is a central use case, but not the only one. Other examples - streaming HTML DOM tree as events for custom transformation into particular LLM input format. Or letting an LLM write a Kaitlin Script to take particular XML and extract only document headers to progressively discover a table of contents without overloading the context window. Similar to jq for JSON. A swiss army knife for streams of meanings. To achieve these goals I have to compromise. I take GFM as a baseline, since it is prevalent in the LLM training data and outputs. I diverge where it would break incrementality of the parser, and LLMs wouldn't output such a thing anyway. I might extend to all the common extensions, as long as they are incremental in nature. But full adherence to particular markdown spec will never be a goal - rather defining a new spec based on this parser, as a baseline markdown format for all the tools (e.g. obsidian-like editor in Golem XIV) we will develop in the future.

[altavir]

OK, I understand. It won't help in my case then.