just discovering that an incremental markdown parsing library doesn't exist, unless 4o is lying to me (likely). going to need this for rendering incremental markdown in dave's ui. it seems geniunely excited and is encouraging me to build this because it hasn't seen it in its training data. LLMs can be adorable at times.
https://chatgpt.com/share/68017a55-dfd4-800f-aabb-ea2abbd282a6
Adorably idiotic ChatGPT lol.
I can’t use this though, its not rust
Oh then just go with Treesitter. Not sure why ChatGPT is objecting to using it. Both typing and appending tokens are just appending some text to a long string and this is exactly what Treesitter is made for. Now, ChatGPT is partially correct that Treesitter may be slightly too powerful for your use case since it’s designed to allow inserting text literally anywhere in the string and you’re only appending but it definitely covers your use case.
I will have to look a the implementation and see if it makes sense. It would be pretty simple to implement it specifically for my use case
The big advantage you get with Treesitter is that it’s trivially easy to add all the other niceties to Markdown. Parsing Markdown itself is relatively simple, it’s the syntax highlighting for the blocks of code and Latex that will get you. Either you have to forgo all of that and content yourself with reading code as plain text or you’ll have to undertake the heroic project of writing your own incremental parser for all of those languages.
also tree sitter isn't technically what I'm trying to do. tree sitter is about maintaining an AST and updating it. I'm not try to do anything fancy like that, I just want a incremental/partially parsed token stream that I can render markdown from.
due to the realtime nature of immediate mode rendering, it would be inefficient to re-parse everything every frame, so a partially parsed data structure that you can continue from makes the most sense.
Exactly, with Treesitter the AST means that you don’t have to completely reparse every token. Treesitter just updates the existing AST rather than restarting from scratch so it’s super efficient.
Does that AST add some complexity? Of course, but it’s the only way to keep a consistent API if you want to have highlighting for code blocks and LaTeX.
