Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 24 18:42
    Konfekt edited #419
  • Oct 24 18:41
    Konfekt opened #419
  • Oct 21 12:32
    atanasj opened #371
  • Oct 20 08:48
    abudden opened #370
  • Sep 27 15:04
    Konfekt closed #418
  • Sep 27 15:04
    Konfekt opened #418
  • Sep 20 14:05
    carldelfin opened #417
  • Sep 11 11:00
    Konfekt closed #415
  • Sep 11 11:00
    Konfekt commented #415
  • Sep 07 19:24

    fmoralesc on master

    python/helpparser: fix version … (compare)

  • Sep 07 19:24
    fmoralesc closed #332
  • Sep 06 14:01
    Konfekt edited #415
  • Sep 06 13:40
    Konfekt opened #416
  • Sep 06 13:37
    Konfekt opened #415
  • Sep 06 13:23
    Konfekt commented #332
  • Sep 06 10:58
    user202729 commented #366
  • Sep 06 10:56
    user202729 commented #366
  • Sep 05 14:19
    lyndhurst commented #368
  • Sep 05 01:37
    user202729 opened #369
  • Sep 05 01:36
    user202729 commented #367
Caleb Maclennan
@alerque
If you can figure out how to pass the current buffer in Lua and describe the exact output you want back, this won't be hard to wire up.
Felipe Morales
@fmoralesc
I will look into it
Caleb Maclennan
@alerque
{ start byte, end byte, tag name } is the info I have so far, do we need more than that? Pass whether the tag is blockwise or inline? Have two functions, one to get a block tag map and a second to get inlines?
Felipe Morales
@fmoralesc
That is absolutely all we need, and probably we should try with the current scheme before anything else. The problem we have to solve now is how to use this data. There is no way to add multiline highlights in the nvim API, so we should try with pushing region definitions (for this we have to calculate line and row of the start and end of regions). If so, we can give a uniform procedure to apply the highlighting.
Another problem will be to decide when to trigger the drawing. If this turns out to be as fast as we seem to expect, maybe we can get away with reapplying the syntax on every document change. If not, we will have to be a bit more clever (then it might be useful to distinguish between block and inline elements).
Caleb Maclennan
@alerque
It's probably worth calculating the line and column numbers on the Rust side. It's going to be an order of magnitude faster I think. But before we start optimizing that some basic proof of concepts are probably in order.
I'm guessing we'll have two refresh loops, one for the block under the cursor that only recalcs inline elements, and one for the whole buffer to find blocks.
Felipe Morales
@fmoralesc
I would imagine piggybacking on vim's line and column calculations is less expensive than doing it on the rust side (vim should have the buffer in memory already).
Caleb Maclennan
@alerque
vps = require("libvim_pandoc_syntax")
offsets = vps.get_offsets(vim.api.nvim_get_current_buf())
for i, event in ipairs(offsets) do
    print(event.tag, event.start)
end
That can be run from viml wrapped in lua << EOF .... EOF.
See just pushed commits.
AFK
Felipe Morales
@fmoralesc
@alerque I also made some progress on the python side, actually managed to get some highlighting working. passing data to python from rust was not fun, eventually i cheated and decided to pass lists of strings instead of something more natural. it looks like the lua api is simpler to manage.
regarding the highlighting itself, there is no support for multiline things at all, so that should be emulated somehow.
Caleb Maclennan
@alerque
Were you using pyo3 or something else? It looks lik it could pass data a similar way.
Felipe Morales
@fmoralesc
Using pyo3. I'm just not used to type systems, so I just went for the quickest hack. Ideally, I would be able to pass a dictionary like {1: ((start, end), tag)}, but that means I need to build a heterogeneous vec!, and I didn't understand how that works.
Caleb Maclennan
@alerque
I don't think Vec is what you want. You probably want a struct for the data we're goung to pass, then a trait to convert it to the right Python data type on the way out. The Lua one needs something like that too actually. What I have is a quick hack to stuff the Lua output table directly but I should have used a Struct with a conversion trait.
In fact with the right struct in place and type coercing either/both Python & Lua interfaces should be able to work from the same Rust code.
Felipe Morales
@fmoralesc
That makes sense.
I really need to learn more about how rust works, as I mentioned before I had never coded in it.
Caleb Maclennan
@alerque
I've spent the last couple weeks reading "the book" and writing a couple projects in it. There is a lot to love!
If we get the vim site sorted out I can figure out how to poss a dictionary.
Felipe Morales
@fmoralesc
Did you see the code I wrote for the vim side?
Caleb Maclennan
@alerque
The interesting thing is if we get this right we should be able to swap out parts and use Pandoc (or commonmark-hs+pandoc extensions when it has them) and get syntax highlighting based on the actual parser.
No sorry I've been very in and out.
Felipe Morales
@fmoralesc
That's ok ;) I think I'm going to bed now, so I'll stop here (or I'll never stop :p)

The interesting thing is if we get this right we should be able to swap out parts and use Pandoc (or commonmark-hs+pandoc extensions when it has them) and get syntax highlighting based on the actual parser.

Yeah, the pulldown-cmark find was really nice because it gave us something to try this idea on.

Caleb Maclennan
@alerque
I see your Python/Rust code now from a Github comment but I don't see any vim site stuff. Where did you post that?
Caleb Maclennan
@alerque
Not to confuse the issue with more architecture options, but here is a tree-sitter for CommonMark. There are tree-sitter syntax highlighters for Neovim.
Felipe Morales
@fmoralesc
I created a private repo, I invited you as a collaborator
Caleb Maclennan
@alerque
So something Neovim HEAD has right now that isn't in the last release is nvim__buf_set_luahl() (docs).
Felipe Morales
@fmoralesc
I haven't been paying much attention to things coming from nvim, that could be interesting ;)
Caleb Maclennan
@alerque
Found it. Github doesn't notify for those except through email, which I've been ignoring today.
Actually no I didn't find it, that was the vim-pandoc org invite!
Felipe Morales
@fmoralesc
XD
Caleb Maclennan
@alerque
There ... that one.
Caleb Maclennan
@alerque
I got something similar going in Lua. At some point I'll rehash it so we can try from Python too using one code base to compile both native modules until we figure out what direction we're going.
I think we need to split the syntax identifier → highlight group -> linked name so that we can feed different parser's data and map it into the same set of groups the legacy rules use.
Felipe Morales
@fmoralesc
Last night I implemented an algorithm to get around the multiline limitations of nvim's add_highlight mechanism.
I think we need to ditch the legacy groups, because of the differences in the "parsing" strategies there won't be a 1:1 mapping anyway.
I also found we don't only need to capture the Start events, we also need to capture the offsets of Code, Html, FootnoteReference, Rule, and maybe HardBreak.
Felipe Morales
@fmoralesc
A weakness of the mechanisms we are using now is that they don't seem to be introspectable: once we push highlighting to the buffer, we cannot retrieve much information about it back (this is something we need for context-aware autoformatting in vim-pandoc, for example). I just discovered that we can retrieve the position of the boundaries of the highlight elements with nvim_buf_get_extmarks(0, ns_id, 0, -1, {}) (this emits output like: [[85, 0, 0], [86, 1, 0], [87, 2, 0], [88, 3, 5], [89, 3, 15], [90, 5, 6], [91, 6, 0], [94, 6, 53], [92, 7, 0], [93, 8, 0], [95, 10 , 0], [96, 11, 0]]). But I can't find a way to retrieve the hlgroups.
Caleb Maclennan
@alerque
We should actually be able ta feed back matched pairs of Start and End data if that's what we need.
Felipe Morales
@fmoralesc
That's actually a good point. But I think the problem above is also an oversight in neovim's API. The underlying 'extmarks' include this information, but it is not exposed.
Caleb Maclennan
@alerque
I agree from all I'm seeing this seems like a deficiency in the API, we shouldn't have to be providing a workaround for this!
Caleb Maclennan
@alerque
Honestly the API is pretty off balance it generally. Why the deuce is there an API equivalent of line2byte() (nvim_buf_get_offset) that you can query with any buffer handle, but you have to rely on running a callback on the viml interface for byte2line() and assume the current buffer isn't something else when you do?
Caleb Maclennan
@alerque
So the current approach is going to need some work. At least from the Lua site. The performance is screaming right now even on a loop running on ever line change event with no delay or backgrounding. But then at some point (after a few thousand lines) it suddenly decides it needs to chunk the process and apply the highlights in batches (or does some garbage collection or who knows what) and the whole UI freezes while it does it's thing for a few seconds.

I also found we don't only need to capture the Start events, we also need to capture the offsets of Code, Html, FootnoteReference, Rule, and maybe HardBreak.

What do you mean by "capture the offsets" here? You mean figure out what is the delimiter for the syntax element and what is the text element inside it?

Caleb Maclennan
@alerque
Nevermind that last question. I see what you mean.
Caleb Maclennan
@alerque
Done. There were a few more too. We'll need more special handling for some of them to extract more detail like how much is delimiter vs. inside content, but one thing at a time.