also tree sitter isn't technically what I'm trying to do. tree sitter is about maintaining an AST and updating it. I'm not try to do anything fancy like that, I just want a incremental/partially parsed token stream that I can render markdown from.
Discussion
due to the realtime nature of immediate mode rendering, it would be inefficient to re-parse everything every frame, so a partially parsed data structure that you can continue from makes the most sense.
Exactly, with Treesitter the AST means that you don’t have to completely reparse every token. Treesitter just updates the existing AST rather than restarting from scratch so it’s super efficient.
Does that AST add some complexity? Of course, but it’s the only way to keep a consistent API if you want to have highlighting for code blocks and LaTeX.