pest-parser/pest

The Elegant Parser

Jesse Jafa

@awareness481

Hello, I'm trying to match a string that starts with one or more #, contains any WHITE_SPACE | ALPHANUMERIC characters & zero to one NEWLINE. But this seems to match more than one newlines...?

comment = {"#"+ ~ (ASCII_ALPHANUMERIC | WHITE_SPACE)+ ~ NEWLINE{0, 1}}

1 reply

Anchor Modeling

@anchormodeling_twitter

@lwandrebeck Do you have any insight in the current development status, as if the project is or will be actively maintained?

Vladimir Uogov

@vulogov

When trying to run samples from book, got a message:

error[E0468]: an extern crate loading macros must be at the crate root
--> src/lang/mod.rs:4:1
|
4 | extern crate pest_derive;
| ^^^^^^^^^^^^^^^^^^^^^^^^^

Laurent Wandrebeck

@lwandrebeck

@anchormodeling_twitter project is active, several releases these last weeks :)

Raphaël Duhen

@HoloTheDrunk

Hi, is there a way to have ~ mean mandatory whitespace instead of optional? I'm making an esolang that's almost only words from a conlang and it's a bit unwieldy to make nearly every rule atomic and then adding WHITESPACE+ between tokens and rules...

Omid Rad

@omid

Hello,
Since I'm new here, let me thank the team for the library <3
And then ask my question...
Is it possible to create pest grammar with macros? Or even dynamically at runtime?

(use case: I would like to let my lib users the ability to add some specific logics to the defined grammar)

William Rågstad

@WilliamRagstad

Hello, I'm new to your community and have started writing a parser for my language (re-implementing from C# to Rust).

I would like to say that I completely fell in love with your pest grammar files with compile-time validation, it's impressive that left recursion can be detected statically.

I wonder if there is any good tooling for formatting pest files via command line using the pest-fmt lib?

William Rågstad

@WilliamRagstad

And a more advanced question, how do I handle operator precedece for user defined operators? Is there a way of injecting new operator tokens? and if so how do I manage their precedence?

James Harton

@jimsynz

Hi folks. I just tried to switch from the old precedence climber to the new pratt parser and am having some issues. It would be great if there was some documentation on how to migrate between them.

chaosprint

@chaosprint:matrix.org

[m]

Hi, is it possible to use no_std at the moment? is there any limitation?

Tomas Tauber

@tomtau

Hello,
Since I'm new here, let me thank the team for the library <3
And then ask my question...
Is it possible to create pest grammar with macros? Or even dynamically at runtime?
(use case: I would like to let my lib users the ability to add some specific logics to the defined grammar)

@omid
it should be possible to some extent -- see e.g. https://github.com/pest-parser/site/blob/master/src/main.rs#L112 which is how the grammars are dynamically compiled on https://pest-parser.github.io/#editor

I wonder if there is any good tooling for formatting pest files via command line using the pest-fmt lib?

@WilliamRagstad
AFAIK pest-fmt itself can be compiled as a command-line tool: https://github.com/pest-parser/pest-fmt/blob/master/src/main.rs ?

Tomas Tauber

@tomtau

Hi folks. I just tried to switch from the old precedence climber to the new pratt parser and am having some issues. It would be great if there was some documentation on how to migrate between them.

@jimsynz you can check out this calculator example which has both the old precedence climber and the new pratt parser tests: https://github.com/pest-parser/pest/blob/master/pest/tests/calculator.rs#L202 Does it help?

Hi, is it possible to use no_std at the moment? is there any limitation?

@chaosprint:matrix.org yes, it's possible. One limitation is that the error type won't have thiserror goodies derived: https://github.com/pest-parser/pest/blob/master/pest/src/error.rs#L28 (because the Error trait isn't in core: https://users.rust-lang.org/t/is-trait-std-error-moving-to-core-error/36443 )

Tomas Tauber

@tomtau

And a more advanced question, how do I handle operator precedece for user defined operators? Is there a way of injecting new operator tokens? and if so how do I manage their precedence?

@WilliamRagstad
https://docs.rs/pest/latest/pest/pratt_parser/struct.PrattParser.html (note that it doesn't support ternary operators)

Tomas Tauber

@tomtau

@/all The pest.rs domain has expired. For the moment, you can use https://pest-parser.github.io to access the website -- if you cannot access the website (i.e. it still redirects to pest.rs), you can try clearing out your browser cache. FYI there's an effort to crowdfund the domain renewal: https://github.com/pest-parser/pest/discussions/717

ethin

@ethindp:the-gdn.net

[m]

Hey guys, trying to write an ada grammar following their BNF syntax as definedhere. In particular, I'm strictly just trying to cover chapter 2 for now (specifically: identifiers; numeric, character, and string literals; comments; pragmas; and reserved words). The specific productions are collected in P.1 of the reference manual (the specific sections within that subclause are sections 2.3-2.8). I was hoping that you guys could help me ensure that my translation of those productions is correct. I'll attach what I have so far (its a bit long and I don't want to make this (already long) message even longer). I'm new to PEGs, so I'll definitely have quite a few more questions before I finish this.

ada.pest

I would definitely like to say that, even though I've just started using pest, I'm really liking it. The only PEG library that rivals it, I think, is PEGTL for C++, but that uses template monstrosities for its grammars, and though they've made it somewhat pleasant to use, the template-ness can definitely be very confusing at times. I imagine taht once I fully get used to it I'll enjoy it even more than I already do

BrokenKeyboard

@InterestingBrainPoops

For the pratt parser, do you need to implement the parser manually? Or can you use the derive macro still

Tomas Tauber

@tomtau

@InterestingBrainPoops you can use the derive macro; you just need to specify the precedence manually. See these new book sections: https://pest-parser.github.io/book/precedence.html
https://pest-parser.github.io/book/examples/calculator.html

The Calculator example is based on the excellent tutorial originally written by @wildarch. For questions or suggestions, please feel free to open issues or submit a PR on https://github.com/pest-parser/book !

Tomas Tauber

@tomtau

@/all The pest.rs domain has expired. For the moment, you can use https://pest-parser.github.io to access the website -- if you cannot access the website (i.e. it still redirects to pest.rs), you can try clearing out your browser cache. FYI there's an effort to crowdfund the domain renewal: https://github.com/pest-parser/pest/discussions/717

It's back up now: https://pest.rs

ocean

@owocean:seapunk.xyz

[m]

im having trouble capturing everything inside of two squiggly brackets.

"{" ~ ANY* ~ "}" does not work

ocean

@owocean:seapunk.xyz

[m]

nvm

ocean

@owocean:seapunk.xyz

[m]

anyone know if theres a way to generate vscode syntax highlighting from pest grammar

Anna Allen

@generic-github-user

Having some trouble with left-recursive rules

The error message recommends using pest::pratt_parser, but it seems to be underdocumented

Anna Allen

@generic-github-user

I would prefer just to rewrite the offending rules if possible

trying to parse a C-like language in which expressions can be called like functions

Julian Scheid

@jscheid_gitlab

Dockerfiles can have a parser directive to specify the line continuation character. It's a backslash by default but can be set to anything else. Is this something that can be modeled with pest, without writing a parser from scratch? I'm thinking a pre-pass to look for the directive with a regex, but how would I pass the character into the grammar?

(pest might be the completely wrong tool for the job here but that's what https://github.com/HewlettPackard/dockerfile-parser-rs/ is using already)

I suppose if all else fails the pre-pass could resolve line continuation, strings and comments and pest could pick up from there but would be nice to avoid that...

Julian Scheid

@jscheid_gitlab

I suppose pest-parser/pest#600 would help with it, but seems like that isn't going anywhere

Julian Scheid

@jscheid_gitlab

Scratch all that, with heredocs as well it looks like a lost cause

Fengda Huang

@phodal

Hello, I try to add LSP (language server protocol) for my DSL, but had issue on fault tolerance. Any ideas to skip error when user in typing ?

orzen

@orzen:matrix.org

[m]

Is the editor on the website using a different parser than pest::Parser? I'm running with identical files(pest and input) locally and on the website and it works on the website but not locally.

orzen

@orzen:matrix.org

[m]

Nvm found it the mistake

Raphaël Duhen

@HoloTheDrunk

@phodal What you're describing doesn't seem like a Pest thing but rather like something that'd be handled by your language server implementation, as a delay between every parse for example. It could even be an editor-level change, but it doesn't make sense to me that the solution would lie with Pest

1 reply

Tomas Tauber

@tomtau

pest v2.5.0 with a new grammar debugger published: https://github.com/pest-parser/pest/discussions/739

Raphaël Duhen

@HoloTheDrunk

Nice!
On that note, how hard do you think it would be to implement a sort of fuzzy test generator based on a Pest grammar file? I'd be willing to help.
One of the biggest issues when writing complex parsers for me is always testing all possible valid inputs to check if the code part handles them properly once the parser has done its thing. As such, something that could go through the rules recursively and generate valid input until it has covered all paths would be a massive help for testing (and maintainability and resilience of the implementation).

Tomas Tauber

@tomtau

@HoloTheDrunk the simplest is just to use e.g. cargo-fuzz to generate arbitrary inputs, run it over long time and check what was covered: https://rust-fuzz.github.io/book/cargo-fuzz/coverage.html If you want only valid inputs, it should be possible... there's this older paper that describes such technique for CFG, and it shouldn't be difficult to adapt it for PEG: https://www.csse.canterbury.ac.nz/research/reports/TechReps/1997/tr_9710.pdf

https://www.fuzzingbook.org/html/Grammars.html

Tomas Tauber

@tomtau

You can see simple_grammar_fuzzer in Python in that book.

Tomas Tauber

@tomtau

@HoloTheDrunk btw, it seems someone has already done something like that: https://github.com/pest-parser/pest/issues/503#issuecomment-835901312 (not for all grammars, but something in that direction) ?

FUSEN

@Fus3n

how can i fix this?

CompareExpr contains Expr but Expr also contains CompareExpr is there any way to fix this error?

Raphaël Duhen

@HoloTheDrunk

@Fus3n as the message says, this is an issue with the precedence of operations within your grammar.
This is the point where you're gonna have to start writing Rust instead of just Pest grammar as precedence shenanigans are handled through code.
Look into the fourth chapter of the book

Raphaël Duhen

@HoloTheDrunk

@tomtau I see, that last one seems to be more or less exactly what I was thinking about, albeit a bit simpler.
I was already planning on starting to look into contributing to Pest recently, so I'll have a look into a possible implementation of a generator tightly couple with the Pest internals in the hopes of being able to handle more patterns.

Where communities thrive

People

Repo info

Activity