@alerque I also made some progress on the python side, actually managed to get some highlighting working. passing data to python from rust was not fun, eventually i cheated and decided to pass lists of strings instead of something more natural. it looks like the lua api is simpler to manage.
regarding the highlighting itself, there is no support for multiline things at all, so that should be emulated somehow.
regarding the highlighting itself, there is no support for multiline things at all, so that should be emulated somehow.
I don't think
Vec is what you want. You probably want a struct for the data we're goung to pass, then a trait to convert it to the right Python data type on the way out. The Lua one needs something like that too actually. What I have is a quick hack to stuff the Lua output table directly but I should have used a Struct with a conversion trait.
In fact with the right struct in place and type coercing either/both Python & Lua interfaces should be able to work from the same Rust code.
I really need to learn more about how rust works, as I mentioned before I had never coded in it.
If we get the vim site sorted out I can figure out how to poss a dictionary.
No sorry I've been very in and out.
The interesting thing is if we get this right we should be able to swap out parts and use Pandoc (or
commonmark-hs+pandocextensions when it has them) and get syntax highlighting based on the actual parser.
Yeah, the pulldown-cmark find was really nice because it gave us something to try this idea on.
Not to confuse the issue with more architecture options, but here is a tree-sitter for CommonMark. There are tree-sitter syntax highlighters for Neovim.
So something Neovim HEAD has right now that isn't in the last release is
nvim__buf_set_luahl() (docs).
Actually no I didn't find it, that was the vim-pandoc org invite!
I think we need to split the syntax identifier → highlight group -> linked name so that we can feed different parser's data and map it into the same set of groups the legacy rules use.
I think we need to ditch the legacy groups, because of the differences in the "parsing" strategies there won't be a 1:1 mapping anyway.
I also found we don't only need to capture the
Start events, we also need to capture the offsets of Code, Html, FootnoteReference, Rule, and maybe HardBreak.
A weakness of the mechanisms we are using now is that they don't seem to be introspectable: once we push highlighting to the buffer, we cannot retrieve much information about it back (this is something we need for context-aware autoformatting in vim-pandoc, for example). I just discovered that we can retrieve the position of the boundaries of the highlight elements with
nvim_buf_get_extmarks(0, ns_id, 0, -1, {}) (this emits output like: [[85, 0, 0], [86, 1, 0], [87, 2, 0], [88, 3, 5], [89, 3, 15], [90, 5, 6], [91, 6, 0], [94, 6, 53], [92, 7, 0], [93, 8, 0], [95, 10 , 0], [96, 11, 0]]). But I can't find a way to retrieve the hlgroups.
Honestly the API is pretty off balance it generally. Why the deuce is there an API equivalent of
line2byte() (nvim_buf_get_offset) that you can query with any buffer handle, but you have to rely on running a callback on the viml interface for byte2line() and assume the current buffer isn't something else when you do?
So the current approach is going to need some work. At least from the Lua site. The performance is screaming right now even on a loop running on ever line change event with no delay or backgrounding. But then at some point (after a few thousand lines) it suddenly decides it needs to chunk the process and apply the highlights in batches (or does some garbage collection or who knows what) and the whole UI freezes while it does it's thing for a few seconds.
I also found we don't only need to capture the
Startevents, we also need to capture the offsets ofCode,Html,FootnoteReference,Rule, and maybeHardBreak.
What do you mean by "capture the offsets" here? You mean figure out what is the delimiter for the syntax element and what is the text element inside it?
I tested the lua highlighting a bit, and it does seem really fast! Sadly, all the documents I'm working on currently use features that the parser does not support :p I also found a document where the highlighting straight up stopped after ~570 lines.
I'll dig into that next time I work on this. I thought of several ways we can improve it. We don't need to clear from the top of the document, only from the line under cursor and tell the event queue to skip us to there. And we can safely stop updating past After the next full block. If we keep a small stack with block boundaries we should be able to process very little data on redraws.
Also we can do things like add a function to straight up ask what kind of block / element we are in at any given cursor point.
That doesn't have to be derived from the syntax parsing at all, we can skip straight to the answer.
I am excited about wiring this up to something with full Pandoc syntax support.
Hey what do you think about the idea of actually splitting this off as a different plugin? I don't see how
pulldown-cmark is going to support all of Pandoc's extentions any time soon which means we could only merge this work as some sort of alternate reality to the default Pandoc mode until we get our hands on commonmark-hs or similar. I'm actually really hopefull the latter will happen soon and/or we can directly wire into it in a similar way. I see the need for a strict CommonMark syntax variant, and maybe the GFM mode too, both of which pulldown-cmark works for now. We could reuse all/part of the design or make it a dependency if there is anything we can gain now.
I'm not sure on this, just brain-storming. Trying to convince myself maybe ;-)