Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Hanns Holger Rutz
    @Sciss
    So no. I think I'm happy with what I get from Metals for now. Just wondering where I get my syntax highlight (parser/lexer) from? Preferably with synchronous code.
    well it sort of depends on how your editor does syntax highlighting
    Hanns Holger Rutz
    @Sciss
    Currently, it just runs the lexer on the whole buffer when you edit, since that is so far a pretty fast operation. Then I get like a token iterator, and that's mapped to font attributes. I already do one extra mapping from scalariform tokens to my own, so that wouldn't be the problem.
    But ideally I could distinguish whether a symbol is a class, an object, refers to a local variable or a class variable, etc. I thought that was called semantic highlight.
    Chris Kipp
    @ckipp:matrix.org
    [m]
    and then that's how you do highlighting currently? mapping the scalariform tokens to your own?
    Hanns Holger Rutz
    @Sciss
    Yes. And on top I do the compiler error highlighting coming from (Scala 2 presentation compiler / now Metals)
    Can I do the highlighting entirely through Metals?
    Chris Kipp
    @ckipp:matrix.org
    [m]
    gotcha
    not currently, no
    out of total curiosity, do you have like a good link explaining your project goals or something? I'd be curious to read more about it
    Hanns Holger Rutz
    @Sciss
    Hmmm. And do you happen to know a good Scala library that would help me here, that supports both Scala 2 and 3?
    Like, are the VIM people writing that lexer in vim-script or whatever it's called? Or is there some Scala code for it?
    Chris Kipp
    @ckipp:matrix.org
    [m]
    I don't know of anything in scala
    I think semantic tokens could really help you here
    sicne then it'd be handled over LSP as well
    Hanns Holger Rutz
    @Sciss
    Yeah. But you say that is not yet implemented?
    Chris Kipp
    @ckipp:matrix.org
    [m]
    have you by chance looked at all into NeoVim and how they've implemented tree-sitter?
    it's quite impressive and incredible fast
    however the Scala grammar is also not complete for it
    Hanns Holger Rutz
    @Sciss
    No. I have refrained so far from looking into scary projects like Vim, Emacs
    Chris Kipp
    @ckipp:matrix.org
    [m]
    and no, but we do have a feature request for it scalameta/metals-feature-requests#57
    scary projects like Vim, Emacs.... You're literally writing your own editor 😆
    Hanns Holger Rutz
    @Sciss
    So how does VS Code do it? With some grammer file and a typescript library?
    Yeah, but it's scope is far more limited than Vim or Emacs :)
    Chris Kipp
    @ckipp:matrix.org
    [m]
    Hanns Holger Rutz
    @Sciss
    thanks, I will look into that. It says it's both Scala 2 and 3, so that's great.
    Chris Kipp
    @ckipp:matrix.org
    [m]
    yea plus this is regex based
    so it's not semantic
    Hanns Holger Rutz
    @Sciss
    Ok. This is why IMHO the Scala 2 presentation compiler was such a great project (in concept). It's a shame that it didn't live on as a public and official API.
    Chris Kipp
    @ckipp:matrix.org
    [m]
    The Scala 3 compiler is also very impressive and imo probably way easier even to work with
    they've done a really great job of having interactivity in mind from the start
    Hanns Holger Rutz
    @Sciss
    Yeah, but I got the message that none of this will be official API, so you'll risk breaking your library once every while if you depend on it.
    Chris Kipp
    @ckipp:matrix.org
    [m]
    if you haven't seen this before, it's a good watch https://www.youtube.com/watch?v=doFQeaY_EdQ
    Hanns Holger Rutz
    @Sciss
    :thumbsup:
    Hanns Holger Rutz
    @Sciss
    since Guillaume mentions (in 2018) ammonite for syntax highlighting, perhaps that's a better option for me than trying to depart from some JavaScript/TS code...
    Chris Kipp
    @ckipp:matrix.org
    [m]
    have you looked at fansi at all?
    I believe under the hood in Amm there is a lexer of sorts that is using fansi to color stuff
    Hanns Holger Rutz
    @Sciss
    Yes, that's just (as far as I see) the last step of converting token symbols to terminal printable colours. In my case, I am going through Java2D / Swing, so I really just need a lexer that can be configured to produce correct results for both Scala 2.13 and 3. I have to read and understand how that regex tree of the VS code project is traversed, and if those regexes in general would be easy to translate to be compatible with Java regex (and if that performs fast or not). I once had a JFlex based grammar - very fast - but it's outdated Scala 2, and I don't think I am capable of writing that for Scala 3 myself. Stuff has changed, e.g. writing numerical literals like 123_456, trailing commas etc., the syntax changed a lot in the last years.
    Hanns Holger Rutz
    @Sciss
    It looks like well this person wrote a textmate bundle in 2017, and then I forked it from here, and pasted stuff over from this version abandoned in 2018, and I tweaked in a bit to support this Scala feature. Would be nice to have some principled and authoritative source that is a bit more concrete than a backus-naur form.
    Chris Kipp
    @ckipp:matrix.org
    [m]
    I mean if you are going to put in a ton of work to implement this, you could just contribute semantic tokens to metals 😸
    Hanns Holger Rutz
    @Sciss
    haha
    I am actually looking for people to help me make this work in the first place. So ideally an existing library I can use, like scalariform used to be.
    Chris Kipp
    @ckipp:matrix.org
    [m]
    Is scalariform still maintained?
    Hanns Holger Rutz
    @Sciss
    It seems to be the fate of tooling libraries for Scala, that they never really work 100%, and are abandoned at random points in their history.
    Last changes are from 2019, and it was used for Scala-IDE (Eclipse) which most definitely is abandoned.
    This was already the schism of Scala 2. It had the presentation compiler, which is the ideal place to give you the lexer. But then over time, the output of the lexer diverged from the actual representation of the source text. Like positions didn't agree, you needed a lot of heuristics to get the "correct" tokens etc.
    Definitely running a bundle of regexes will most likely not get you a 100% correct token stream for Scala.