Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 25 20:23
    a-torgovitsky commented #306
  • Nov 25 19:24
    hgvhgv commented #237
  • Nov 17 22:47
    elshize opened #347
  • Nov 17 20:36
    p3palazzo commented #290
  • Nov 12 14:20
    yushangakki opened #392
  • Nov 07 14:20
    twsh closed #391
  • Nov 07 14:20
    twsh commented #391
  • Nov 07 14:19

    fmoralesc on master

    hypertext: fall back to implici… (compare)

  • Nov 07 14:19
    fmoralesc closed #390
  • Nov 07 14:19
    fmoralesc commented #390
  • Nov 07 14:15
    fmoralesc commented #391
  • Nov 07 14:14
    fmoralesc commented #391
  • Nov 07 13:21
    twsh edited #391
  • Nov 07 13:17
    twsh opened #391
  • Nov 06 23:00
    Chris-Slade opened #390
  • Nov 03 22:44
    hgvhgv commented #237
  • Nov 03 21:53
    Konfekt commented #237
  • Nov 03 13:14
    JanSurft edited #346
  • Nov 03 13:10
    JanSurft edited #346
  • Nov 03 13:10
    JanSurft edited #346
Felipe Morales
@fmoralesc
I really need to learn more about how rust works, as I mentioned before I had never coded in it.
Caleb Maclennan
@alerque
I've spent the last couple weeks reading "the book" and writing a couple projects in it. There is a lot to love!
If we get the vim site sorted out I can figure out how to poss a dictionary.
Felipe Morales
@fmoralesc
Did you see the code I wrote for the vim side?
Caleb Maclennan
@alerque
The interesting thing is if we get this right we should be able to swap out parts and use Pandoc (or commonmark-hs+pandoc extensions when it has them) and get syntax highlighting based on the actual parser.
No sorry I've been very in and out.
Felipe Morales
@fmoralesc
That's ok ;) I think I'm going to bed now, so I'll stop here (or I'll never stop :p)

The interesting thing is if we get this right we should be able to swap out parts and use Pandoc (or commonmark-hs+pandoc extensions when it has them) and get syntax highlighting based on the actual parser.

Yeah, the pulldown-cmark find was really nice because it gave us something to try this idea on.

Caleb Maclennan
@alerque
I see your Python/Rust code now from a Github comment but I don't see any vim site stuff. Where did you post that?
Caleb Maclennan
@alerque
Not to confuse the issue with more architecture options, but here is a tree-sitter for CommonMark. There are tree-sitter syntax highlighters for Neovim.
Felipe Morales
@fmoralesc
I created a private repo, I invited you as a collaborator
Caleb Maclennan
@alerque
So something Neovim HEAD has right now that isn't in the last release is nvim__buf_set_luahl() (docs).
Felipe Morales
@fmoralesc
I haven't been paying much attention to things coming from nvim, that could be interesting ;)
Caleb Maclennan
@alerque
Found it. Github doesn't notify for those except through email, which I've been ignoring today.
Actually no I didn't find it, that was the vim-pandoc org invite!
Felipe Morales
@fmoralesc
XD
Caleb Maclennan
@alerque
There ... that one.
Caleb Maclennan
@alerque
I got something similar going in Lua. At some point I'll rehash it so we can try from Python too using one code base to compile both native modules until we figure out what direction we're going.
I think we need to split the syntax identifier → highlight group -> linked name so that we can feed different parser's data and map it into the same set of groups the legacy rules use.
Felipe Morales
@fmoralesc
Last night I implemented an algorithm to get around the multiline limitations of nvim's add_highlight mechanism.
I think we need to ditch the legacy groups, because of the differences in the "parsing" strategies there won't be a 1:1 mapping anyway.
I also found we don't only need to capture the Start events, we also need to capture the offsets of Code, Html, FootnoteReference, Rule, and maybe HardBreak.
Felipe Morales
@fmoralesc
A weakness of the mechanisms we are using now is that they don't seem to be introspectable: once we push highlighting to the buffer, we cannot retrieve much information about it back (this is something we need for context-aware autoformatting in vim-pandoc, for example). I just discovered that we can retrieve the position of the boundaries of the highlight elements with nvim_buf_get_extmarks(0, ns_id, 0, -1, {}) (this emits output like: [[85, 0, 0], [86, 1, 0], [87, 2, 0], [88, 3, 5], [89, 3, 15], [90, 5, 6], [91, 6, 0], [94, 6, 53], [92, 7, 0], [93, 8, 0], [95, 10 , 0], [96, 11, 0]]). But I can't find a way to retrieve the hlgroups.
Caleb Maclennan
@alerque
We should actually be able ta feed back matched pairs of Start and End data if that's what we need.
Felipe Morales
@fmoralesc
That's actually a good point. But I think the problem above is also an oversight in neovim's API. The underlying 'extmarks' include this information, but it is not exposed.
Caleb Maclennan
@alerque
I agree from all I'm seeing this seems like a deficiency in the API, we shouldn't have to be providing a workaround for this!
Caleb Maclennan
@alerque
Honestly the API is pretty off balance it generally. Why the deuce is there an API equivalent of line2byte() (nvim_buf_get_offset) that you can query with any buffer handle, but you have to rely on running a callback on the viml interface for byte2line() and assume the current buffer isn't something else when you do?
Caleb Maclennan
@alerque
So the current approach is going to need some work. At least from the Lua site. The performance is screaming right now even on a loop running on ever line change event with no delay or backgrounding. But then at some point (after a few thousand lines) it suddenly decides it needs to chunk the process and apply the highlights in batches (or does some garbage collection or who knows what) and the whole UI freezes while it does it's thing for a few seconds.

I also found we don't only need to capture the Start events, we also need to capture the offsets of Code, Html, FootnoteReference, Rule, and maybe HardBreak.

What do you mean by "capture the offsets" here? You mean figure out what is the delimiter for the syntax element and what is the text element inside it?

Caleb Maclennan
@alerque
Nevermind that last question. I see what you mean.
Caleb Maclennan
@alerque
Done. There were a few more too. We'll need more special handling for some of them to extract more detail like how much is delimiter vs. inside content, but one thing at a time.
Felipe Morales
@fmoralesc
I just saw your work, and it looks really great so far. I fixed the python module so it's more or less up to date with the lua module, I'll share my changes in the PR (not large enough to justify a PR on their own).
I tested the lua highlighting a bit, and it does seem really fast! Sadly, all the documents I'm working on currently use features that the parser does not support :p I also found a document where the highlighting straight up stopped after ~570 lines.
Felipe Morales
@fmoralesc
In the python API there was a procedure called update_highlights() that takes a list of highlights and applies them, supposedly to improve redraws. Take a look at how it's implemented:
Caleb Maclennan
@alerque
I'll dig into that next time I work on this. I thought of several ways we can improve it. We don't need to clear from the top of the document, only from the line under cursor and tell the event queue to skip us to there. And we can safely stop updating past After the next full block. If we keep a small stack with block boundaries we should be able to process very little data on redraws.
Also we can do things like add a function to straight up ask what kind of block / element we are in at any given cursor point.
That doesn't have to be derived from the syntax parsing at all, we can skip straight to the answer.
I am excited about wiring this up to something with full Pandoc syntax support.
Felipe Morales
@fmoralesc
Me too! I think this has been a productive couple of days on this front, the pulldown-cmark find was the key element we needed.
Caleb Maclennan
@alerque
Hey what do you think about the idea of actually splitting this off as a different plugin? I don't see how pulldown-cmark is going to support all of Pandoc's extentions any time soon which means we could only merge this work as some sort of alternate reality to the default Pandoc mode until we get our hands on commonmark-hs or similar. I'm actually really hopefull the latter will happen soon and/or we can directly wire into it in a similar way. I see the need for a strict CommonMark syntax variant, and maybe the GFM mode too, both of which pulldown-cmark works for now. We could reuse all/part of the design or make it a dependency if there is anything we can gain now.
I'm not sure on this, just brain-storming. Trying to convince myself maybe ;-)
Caleb Maclennan
@alerque
...I actually headed down that road in a new repo. I'm open to redirection though, I'm not super confident an independent plugin is the right approach, but it seemed like the most likely way to get something usable out there without getting blocked on Pandoc extension support.
Felipe Morales
@fmoralesc
That sounds fine to me ;) I was thinking we could split the rust library from the plugin.
Caleb Maclennan
@alerque
I thought about that, but I think it would make more sense if we were using it as an RPC end-point. Right now it's pretty closely tied to what we want to call through the vim interface code.
I may have another use for a Lua library wrapping pulldown-cmark so that might make sense ultimately, but I'm not sure yet where to draw the boundary between the various API levels. Since pulldown-cmark already is an idiomatic Rust interface, I'm not sure a 100% Lua wrapper around that is an improvement over a customized wrapper that does exactly what the Lua endpoint needs.
Felipe Morales
@fmoralesc
Sure.
Perhaps one idea for vim-pandoc-syntax could be to use this as a base and override the highlighting for unrecognized block types in some way. For example, I make use of yaml blocks, and this breaks them.
Caleb Maclennan
@alerque
I could see that working.