Currently, it just runs the lexer on the whole buffer when you edit, since that is so far a pretty fast operation. Then I get like a token iterator, and that's mapped to font attributes. I already do one extra mapping from scalariform tokens to my own, so that wouldn't be the problem.
But ideally I could distinguish whether a symbol is a
class, an object, refers to a local variable or a class variable, etc. I thought that was called semantic highlight.
and then that's how you do highlighting currently? mapping the scalariform tokens to your own?
Can I do the highlighting entirely through Metals?
not currently, no
out of total curiosity, do you have like a good link explaining your project goals or something? I'd be curious to read more about it
Like, are the VIM people writing that lexer in vim-script or whatever it's called? Or is there some Scala code for it?
I think semantic tokens could really help you here
sicne then it'd be handled over LSP as well
have you by chance looked at all into NeoVim and how they've implemented tree-sitter?
it's quite impressive and incredible fast
however the Scala grammar is also not complete for it
scary projects like Vim, Emacs.... You're literally writing your own editor 😆
Yeah, but it's scope is far more limited than Vim or Emacs :)
so it's not semantic
The Scala 3 compiler is also very impressive and imo probably way easier even to work with
they've done a really great job of having interactivity in mind from the start
I believe under the hood in Amm there is a lexer of sorts that is using fansi to color stuff
Yes, that's just (as far as I see) the last step of converting token symbols to terminal printable colours. In my case, I am going through Java2D / Swing, so I really just need a lexer that can be configured to produce correct results for both Scala 2.13 and 3. I have to read and understand how that regex tree of the VS code project is traversed, and if those regexes in general would be easy to translate to be compatible with Java regex (and if that performs fast or not). I once had a JFlex based grammar - very fast - but it's outdated Scala 2, and I don't think I am capable of writing that for Scala 3 myself. Stuff has changed, e.g. writing numerical literals like
123_456, trailing commas etc., the syntax changed a lot in the last years.
It looks like well this person wrote a textmate bundle in 2017, and then I forked it from here, and pasted stuff over from this version abandoned in 2018, and I tweaked in a bit to support this Scala feature. Would be nice to have some principled and authoritative source that is a bit more concrete than a backus-naur form.
I mean if you are going to put in a ton of work to implement this, you could just contribute semantic tokens to metals 😸
I am actually looking for people to help me make this work in the first place. So ideally an existing library I can use, like scalariform used to be.
Last changes are from 2019, and it was used for Scala-IDE (Eclipse) which most definitely is abandoned.
This was already the schism of Scala 2. It had the presentation compiler, which is the ideal place to give you the lexer. But then over time, the output of the lexer diverged from the actual representation of the source text. Like positions didn't agree, you needed a lot of heuristics to get the "correct" tokens etc.
Definitely running a bundle of regexes will most likely not get you a 100% correct token stream for Scala.
Just to follow up on the lexer, the tm4s project's core package seems to have a working text mate grammar parser based on joni, a java port of Oniguruma, the reg-ex parser needed for textmate:
val registry = new Registry()
val grammar = registry.loadGrammarFromPathSync("Scala.tmLanguage.json",
TextMateTest.getClass.getResourceAsStream("/Scala.tmLanguage.json"))
val source = "object Foo { def main(args: Array[String]): Unit = println(\"Hello\") }"
val lineTokens = grammar.tokenizeLine(source)
val tokens = lineTokens.getTokens
tokens.foreach { tk =>
val sub = source.substring(tk.getStartIndex, tk.getEndIndex)
println(s"[${tk.getStartIndex} to ${tk.getEndIndex}] source '$sub' has scopes ${tk.getScopes}")
}[0 to 6] source 'object' has scopes [source.scala, keyword.declaration.scala]
[6 to 7] source ' ' has scopes [source.scala]
[7 to 10] source 'Foo' has scopes [source.scala, entity.name.class.declaration]
[10 to 11] source ' ' has scopes [source.scala]
[11 to 12] source '{' has scopes [source.scala, punctuation.section.block.begin.scala]
[12 to 13] source ' ' has scopes [source.scala]
[13 to 16] source 'def' has scopes [source.scala, keyword.declaration.scala]
[16 to 17] source ' ' has scopes [source.scala]
[17 to 21] source 'main' has scopes [source.scala, entity.name.function.declaration]
[21 to 22] source '(' has scopes [source.scala, meta.bracket.scala]
[22 to 26] source 'args' has scopes [source.scala, variable.parameter.scala]
[26 to 27] source ':' has scopes [source.scala, meta.colon.scala]
[27 to 28] source ' ' has scopes [source.scala]
[28 to 33] source 'Array' has scopes [source.scala, entity.name.class]
[33 to 34] source '[' has scopes [source.scala, meta.bracket.scala]
[34 to 40] source 'String' has scopes [source.scala, entity.name.class]
[40 to 41] source ']' has scopes [source.scala, meta.bracket.scala]
[41 to 42] source ')' has scopes [source.scala, meta.bracket.scala]
[42 to 43] source ':' has scopes [source.scala, keyword.operator.scala]
[43 to 44] source ' ' has scopes [source.scala]
[44 to 48] source 'Unit' has scopes [source.scala, entity.name.class]
[48 to 49] source ' ' has scopes [source.scala]
[49 to 50] source '=' has scopes [source.scala, keyword.operator.comparison.scala]
[50 to 58] source ' println' has scopes [source.scala]
[58 to 59] source '(' has scopes [source.scala, meta.bracket.scala]
[59 to 60] source '"' has scopes [source.scala, string.quoted.double.scala, punctuation.definition.string.begin.scala]
[60 to 65] source 'Hello' has scopes [source.scala, string.quoted.double.scala]
[65 to 66] source '"' has scopes [source.scala, string.quoted.double.scala, punctuation.definition.string.end.scala]
[66 to 67] source ')' has scopes [source.scala, meta.bracket.scala]
[67 to 68] source ' ' has scopes [source.scala]
[68 to 69] source '}' has scopes [source.scala, punctuation.section.block.end.scala]
if metals was distributed as a fat jar then it would weigh hundreds of megabytes to support all scala versions