Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 13:31
    SirJosh3917 commented #427
  • 11:50
    zsytssk commented #509
  • 06:23
    baoyachi commented #509
  • 01:37
    zsytssk commented #509
  • Apr 12 21:53
    notriddle commented #498
  • Apr 12 21:47

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • Apr 12 21:38

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • Apr 12 21:31

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • Apr 12 21:30
    notriddle commented #498
  • Apr 12 19:46
    CAD97 commented #498
  • Apr 12 19:38

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • Apr 12 19:38
    CAD97 commented #498
  • Apr 12 18:13

    bors[bot] on staging.tmp

    [ci skip][skip ci][skip netlify] (compare)

  • Apr 12 18:13
    CAD97 commented #498
  • Apr 12 10:07
    bors[bot] closed #473
  • Apr 12 10:07

    bors[bot] on master

    Added lister optimization: redu… Added case in factorizer when r… Added case in factorizer when l… and 14 more (compare)

  • Apr 12 10:07
    bors[bot] commented #473
  • Apr 12 10:05

    bors[bot] on staging.tmp

    (compare)

  • Apr 12 10:05

    bors[bot] on staging

    Added lister optimization: redu… Added case in factorizer when r… Added case in factorizer when l… and 14 more (compare)

  • Apr 12 10:04

    bors[bot] on staging.tmp

    Added lister optimization: redu… Added case in factorizer when r… Added case in factorizer when l… and 14 more (compare)

rtyler
@rtyler:matrix.org
[m]
I am feeling a bit silly, there's got to be a simpler pattern for doing order-independent parsing rules here
I have three rules which can match inside of a block, and I just want the user to be able to order them in either which way
rtyler
@rtyler:matrix.org
[m]
perhaps, worth just asking
Ahmed Abd El Mawgood
@oddcoder
does anyone have any idea how to create custom tokenizer for pest ?

I am dealing with terrible grammer that goes like this

When an io_here token has been recognized by the grammar (see Shell Grammar), one or more of the subsequent lines immediately following the next NEWLINE token form the body of one or more here-documents and shall be parsed according to the rules of Here-Document.

The rules itself are also terrible
like the io_here token is nested multiple layers deep under the layer where we have the NEWLINE stated
xaviex
@xaviex
Hey! Is it possible to have more than one Grammar in a module? (Eg. any way to rename the Rule enum so that they do not conflict)
rtyler
@rtyler:matrix.org
[m]
I could not find a way to put my two grammars in the same module, but putting them in two adjacent modules worked fine
the derive macro brings some things in, like Rule which you mention, which will conflict
Inky
@Inky-developer

Hey, I am having big trouble with my grammar for a custom language. This is a very simplified version:

program = {SOI ~ item* ~ EOI}
item = {block ~ ";"}
block = {"{"~item*~expr?~"}"}
expr = {block}

As a bit of context, an item could be some statement, for example let a = 1; or, as in this grammar, a block: {...};. However, a block can have as its last element an expression (implicite return) and a block can also be interpreted as an expression. This leads to exponential time for inputs that look like this: {{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{.
I know that this is parsed so slowly, because the parser has to backtrack for every {, leading to a 2^n time where n is the number of brackets.
I am quite desperate, is there any way to fix this?

Dragoș Tiselice
@dragostis

@Inky-developer, could you please try pest-parser/pest#473 to see if it improves the situation?

Basically these are warts in the optimiser that's currently not that well designed. I have changes in mind for pest3 that will render these cases obsolete.

Until then, if the patch I sent you doesn't work, you can always get around these cases by using the lookaheads. (& or !)
Dragoș Tiselice
@dragostis
Or you could write something likeblock = {"{"~ block ~ (";" ~ block)* ~ "}"} then disambiguate higher up.
Inky
@Inky-developer
The pull request does unfortunately not prevent the time blowup, on my computer a string with 20 curly braces gets parsed in 46ms, but a string with 27 curly braces already takes 6.1 seconds.
I will check if I can get my grammar to work with the lookahead features, thanks for the suggestion!
If you don't mind, could you please elaborate on your last suggestion?
Dragoș Tiselice
@dragostis
You won't have item/expr productions in your code above, only blocks. You'd. have to disambiguate between them in Rust code.
PaulChernoch-Shell
@PaulChernoch-Shell
I have been using Pest for a few weeks. I am porting a Peg.js grammar to Pest. It is going surprisingly well. My pet project is to write a parser and execution engine for the DMN Feel language in Rust, not a trivial exercise (and not a trivial grammar!) I have a recognizer working for the full grammar. Next up is walking the Pairs and constructing Rust Closures out of the parse tree. Before I do that, I needed to implement a smaller parser to handle Duration strings. They follow the XPATH standard for YearMonth or DayTime. This evening I wrote a nice grammar to parse them:
// Pest Grammar for parsing XPATH Duration strings like this: P2Y3M1DT5H2M5.5S
//   - A "T" divides the year, month, day fields from the hour, minute, second fields.
//   - Some parts may be omitted, but at least one must be specified.
//     For example, PT10M is ten minutes.
//   - If hour, minute and second are all omitted, you must omit the "T".
//   - Never omit the "P".
//   - Only the Seconds field may have a fraction, the others are integers.
//   - A leading minus sign indicates a negative duration. It must come befor ethe "P".
//     For example: -P2Y is negative two years.
//   - Months can be more than twelve, hours more than 24, etc.
date_time = @{
    SOI ~ sign? ~ "P" ~ (
      date ~ "T" ~ time
    | date
    | "T" ~ time
    ) ~ EOI
}
date = {
    year ~ month ~ day
  | year ~ month
  | year ~ day
  | year
  | month ~ day
  | month
  | day
}
time = { 
    hour ~ minute ~ second
  | hour ~ minute
  | hour ~ second
  | hour
  | minute ~ second
  | minute
  | second
}
sign = !{ "-" }
year = !{ number ~ "Y" }
month = !{ number ~ "M" }
day = !{ number ~ "D" }
hour = !{ number ~ "H" }
minute = !{ number ~ "M" }
second = ${ number ~ "S" }
digits = @{ ASCII_DIGIT+ }
number = @{ digits ~ fraction? }
fraction = @{ "." ~ digits ~ !integral }
integral = { "Y" | "M" | "D" | "H" }
PaulChernoch-Shell
@PaulChernoch-Shell
The tricky part was that I wanted to use the wildcard "?", but I need to have at least one of hour, minute, second in the second half in order to include the "T". Thus I had to explicitly list all permutations. I tried to use Rust Regexes, but they have limits not found in C#, Perl or Java implementaions of Regex. So Pest was the only easy way.
michael
@iptq
Hi! I've been having a great time using Pest so far, but I've recently run into a problem; in IMAP, the literal rule is defined as { "{" ~ number ~ "}" ~ CRLF ~ char* }, where number describes how many characters follows the CRLF. Since this is the only thing in the grammar that behaves this way, I was wondering if there was a "backdoor" Pest could offer me,
but all I found was pest-parser/pest#119 , from around 4 years ago. I ended up writing a rather awkward hack to let me do what I wanted, but is the stance of the maintainers still to avoid allowing arbitrary calls in grammar rules?
Specifically, the patch I made (found iptq/pest@4acf395) would let me specify a rule like hello = { #crate::world } where i define fn world(Box<ParserState<Rule>>) -> ParseResult<Box<ParserState<Rule>>> myself. I then added an API that would let me introspect on the current state of the parser
and view the tokens that have been previously parsed. I know it's rather clumsy, but I'm also not too thrilled about redoing my parser from scratch using a different scheme in order to get this single rule to work. I really hope there's a plan to allow something like this in the future since many other proc-macro crates (serde, structopt) let you specify custom
functions to replace core functionality.
MaxBondABE
@MaxBondABE
@dragostis Is there a path where I/others could become members of the organization and take a more active role in Pest? I know I've only made 1 contribution and so would understand if you'd like to see more from me before you're comfortable with that, but I'm wondering if it's a possibility. I really care about this language and would like to see it continue to grow. Specifically I'd like to help finish the book and publish a new release.
Dragoș Tiselice
@dragostis
@MaxBondABE, I'm very happy to hear that. Would anyone else be interested as well? We could potentially make a pest3 WG where I could share more ideas and help move this forward if anyone is willing to put some effort.
michael
@iptq
yeah i would be interested in getting a little bit more involved in my free time, with expanding the book and looking at rfcs/issues
MaxBondABE
@MaxBondABE
I'd propose creating an issue for this, and writing to TWiR to get a call for contribution in the next issue. I have a good amount of free time for the foreseeable.
Dragoș Tiselice
@dragostis
That sounds great!
MaxBondABE
@MaxBondABE
Alright, I'll get started on that.
MaxBondABE
@MaxBondABE
I started the issue here: pest-parser/pest#496
The guidelines for TWiR ask that we submit a CfP with both an issue and a clear set of tasks, so I'll wait until we've discussed the matter further to submit the CfP
Joseph Delgado
@veryjos
question, i can't test my grammar because of "use of undeclared type Rule"
how can i import Rule into my test?
#[derive(pest_derive::Parser)]
#[grammar = "my_grammar.pest"]
pub struct MyParser;

#[cfg(test)]
mod tests {
    use pest::Parser;
    use crate::{MyParser};

    #[test]
    fn some_test() {
        let result = MyParser::parse(Rule::some_rule, input).unwrap();
    }
}
i saw some similar issues after i searched but they're all dead links
Joseph Delgado
@veryjos
nvm it seems to be working if i just import Rule from super, for some reason rust-analyzer didn't like it though
michael
@iptq
Rule is generated by the proc-macro, unfortunately invisibly, so yeah just importing it from wherever ur struct lives should get it to work
Tirth Jain
@hedonhermdev
Can I convert a Pest tree back to source code?
Evan Carroll
@EvanCarroll
I just wanted to give everyone a heads up, I added the tag for pest to StackOverflow so it has it's own tag now. I'm also learning Pest now, and I'm kind of a pita when I learn stuff and I ask a lot of questions, so if you're on StackOverflow and you can help out watch the tag. ;)
I'll probably ask 10 questions myself in the next few days.
I'll also self-answer them if I figure it. So I try to maintain my own questions on the site. Here is my newest question if anyone knows it: https://stackoverflow.com/q/66728220/124486
Victor Sergienko
@singalen
Evan, great! Will go support the tag now.
I actually have a question similar to Tirth's. What is a good way to get an AST from pest now?
pest-consume looks like the most active option. OTOH, there is pest-ast, which, OTTH, seems to be hanging on pest 3 for two years now.
Maybe there is a third option? (except for creating the classes and building the tree by hand)
Sean Young
@seanyoung
I've started working on a peg parser generator which allows left recursive rules and uses ratpack memoization for performance: https://github.com/seanyoung/lrpeg
francesco.cattoglio
@francesco.cattoglio:matrix.org
[m]
Hello there. I just joined the room to say a BIG THANK YOU to everyone involved with pest. I was in dire need of being able to parse some input from my user and I had zero knowledge about lexers and parsing. After spending one week of toying and hacking things together with pest I now have a pretty solid piece of code that does the job.
If I find some time to spare and my colleagues are ok with it, I will create an example project that showcases a few things that I learned along, since every project can always use more tutorials and more examples!
MouseProducedGames
@MouseProducedGames
Is it ok if my question addresses the unofficial pest_consume, rather than pest directly?