sirthias/parboiled2

A macro-based PEG parser generator for Scala 2.10+

vonchav

@voonchav_gitlab

Hi all, apologies if similar questions have already been asked (most likely). What is the recommended testing strategy to test against the parser you've written? I wrote a parser. It's quite nice and I make the grammar quite flexible, especially when dealing with whitespaces. However, that creates a challenge for writing tests to cover all/most cases. I find the tasks of having a decent coverage quite daunting. I thought about writing property tests and Gen's for ScalaCheck. But writing Gen's based off my own grammar is still quite manual - basically translating my syntax to Gen's bit by bit.... Just wondering any advice from the community. Thank you so much.

Boris V.Kuznetsov

@tampler

@voonchav_gitlab Your testing strategy is to write tests for everything you wanna parse. With Zio Test, you can do the following:

import zio.test._
import zio.test.Assertion._

import org.parboiled2.ParseError

object ParserTests {
  val str0 =
    """node Regbuffer : 
  module Shit :""".stripMargin

  val stringSuite = suite("Simple string suite")(
    test("Node Module parsing") {
      val par = new MyParser(str0)
      val res = par.start.run()
      val out = res.fold(ex => par.formatError(ex.asInstanceOf[ParseError]), v => v)
      assert(out, equalTo(Vector(NodeAST("Regbuffer"), ModuleAST("Shit"))))
    })

And easily write small tests in a suite for every aspect you wanna cover in your parser. Than combine them all to parse more complex grammars. Parboiled2 docs have enough explanations and examples on that. Highly recommended to read

vonchav

@voonchav_gitlab

Thanks @tampler. It's interesting that you gave a zio-test as an example. I do use zio and wrote zio-test. But I didn't think I'd use zio-test for parboiled tests :) Yeah, I suppose I'd have to still manually write tests.... I'm interested in what you described, specifically the "combine them all to parse more complex grammars"... did you mean composing small tests into a bigger test? That is intriguing. Would you mind posting an example here? Also, can you post a link to the Parboiled2 docs that show testing? I read through the entire Parboiled2 readme markdown multiple times and some of its example source code. But not so much in regards to writing unit tests. Please point me to the right direction. Thanks again.

Boris V.Kuznetsov

@tampler

Hello. I need to parse a key-value grammar with optional tags. Here's example: tag key0 : value0, key1 : value1. However, the output AST Node depends on whether or not I had a tag for that key. Here's my impl:

def flip: Rule1[Boolean] = rule {
    capture(Word) ~ ws ~> (
      (s: String) => test(s == "tag0") ~ push(true)
    )
  }
  def record: Rule1[BusAST] = rule {
    flip ~ capture(Word) ~ Col ~ bus ~ ws ~> (
      (
        check: Boolean,
        name: String,
        tpe: String,
        len: Int
      ) => BusAST(if (check) Out else In, name, tpe, len)
    )
  }

This impl fails on the second part key1 : value1. I also tried this:

def record: Rule1[BusAST] = rule {
    optional(atomic("tag0")) ~ ws ~ capture(Word) ~ Col ~ bus ~ ws ~> (
      (
        _,
        name: String,
        tpe: String,
        len: Int
      ) => BusAST(if (true) Out else In, name, tpe, len)
    )
  }

The later parses my input fine, but I can't instantiate the correct AST type since I'm missing the knowledge of whether or not I had "tag0" there. @alexander-myltsev any advice? Thanks

Alexander Myltsev

@alexander-myltsev

let me know if that works for you:

def record: Rule1[BusAST] = rule {
    optional(atomic(capture("tag0”))) ~ ws ~ capture(Word) ~ Col ~ bus ~ ws ~> (
      (
        tagOpt: Option[String],
        name: String,
        tpe: String,
        len: Int
      ) => BusAST(if (tagOpt.isDefined) Out else In, name, tpe, len)
    )
  }

Boris V.Kuznetsov

@tampler

@alexander-myltsev Perfecto ! Worked like a charm, thank you!

Alexander Myltsev

@alexander-myltsev

also, you could try to cut the corner:

def record: Rule1[BusAST] = rule {
  (test("tag0") ~ push(Out) | push(In))  ~ ws ~ capture(Word) ~ Col ~ bus ~ ws ~> BusAST 
}

but I’m not sure if (test("tag0") ~ push(Out) | push(In)) passes the typer and derives the correct least supertype of In and Out. you should check on your side

Boris V.Kuznetsov

@tampler

The prev solution looks simpler and delivers a better logical sense. Thanks for your support, Parboiled2 is awesome !

Boris V.Kuznetsov

@tampler

@alexander-myltsev For the sake of science, I tried the second option as :

 def record: Rule1[BusAST] = rule {
    test(true) ~ push(Out) | push(In) ~ ws ~ capture(Word) ~ Col ~ bus ~ ws ~> BusAST
  }

2 issues with this option:

I had to put test(true) to make this compile, since test requires Boolean on input, not String

Still getting a compile error:

{
found   : org.parboiled2.Rule[shapeless.HNil,Product with Serializable :: shapeless.HNil]
required: org.parboiled2.Rule[shapeless.HNil,chipa.parser.ast.BusAST :: shapeless.HNil]"
}

Any ideas how to fix ?

Alexander Myltsev

@alexander-myltsev

ah, sorry. test there is wrong. try this one:

def record: Rule1[BusAST] = rule {
  ("tag0" ~ push(Out) | push(In))  ~ ws ~ capture(Word) ~ Col ~ bus ~ ws ~> BusAST 
}

the wrong thing in your solution is that you don’t take into account the precedance of | and ~ operators. ~ has higher one. that means that you’re making two alternatives:
test(true) ~ push(Out) and push(In) ~ ws ~ capture(Word) ~ Col ~ bus ~ ws ~> BusAST. and apparently the least common supertype of Out and BusAST is Product with Serializable

Boris V.Kuznetsov

@tampler

@alexander-myltsev Thank you for this fix! It indeed looks simpler and more Parboiled-style. I used that pattern to scrap my boilerplate a lot. Now parsing a custom language with a 120 lines-of-code parser :D . Thanks to you and Parboiled2 !

Tieru

@Tieru

Hello. Is there any way to split a big class with multiple rules to smaller classes?

Could you provide me an example with such case? I couldn't find any by myself

Alexander Myltsev

@alexander-myltsev

hi @Tieru

take a look at that project parser https://github.com/sangria-graphql/sangria/blob/master/src/main/scala/sangria/parser/QueryParser.scala

Tieru

@Tieru

So the solution is using traits? Or I miss something else?

Alexander Myltsev

@alexander-myltsev

Wow, missed your question, sorry

Yes, trait is an answer

Tieru

@Tieru

Thanks. It worked :)

Travis Brown

@travisbrown

Has anyone tried porting parboiled2 to Dotty? I took a stab at it this weekend and have support for ~80% of its functionality, with a focus on what http4s needs, but before I put more time into it I want to make sure I'm not duplicating effort.

vonchav

@voonchav_gitlab

OMG, Travis, you're a saint. I know the project is in good hands when you get involved. I use Parboiled2. I'd love that it's Dotty ready.

Boris V.Kuznetsov

@tampler

@travisbrown do you need asap? I also want to work this with Dotty, but the later doesn't work well with Shapeless, which is a transient dep here

Shapeless 3.0.0 is under dev. I think We'll need to port Parboiled2 to Parboiled3 first :D

Travis Brown

@travisbrown

@tampler Current status is here: https://github.com/sirthias/parboiled2/issues/210#issuecomment-597505806

@tampler There's no rush on this from my perspective, since I have it migrated enough for what I needed this week. The subset of Shapeless that Parboiled uses is small enough that it's not much of an issue.

Didac

@umbreak

I have a parsing condition to satisfy which is

it matches if the sequence starts with '@' and it follows with one or more characters (as long as some are NOT alpha numeric), or if the sequence does not start with '@'

So basically,
matched: ‘one’, ’@one.two’
not matched: ‘@one'

Didac

@umbreak

So I have this:

  def term: Rule0 = rule { termMatch1 | termMatch2 }
  private def termMatch1: Rule0 = rule { '@' ~ oneOrMore(All) }
  private def termMatch2: Rule0 = rule { oneOrMore(AlphaNum) }

But I have to exclude terms that match

private def termExclude: Rule0 = rule { '@' ~ oneOrMore(AlphaNum) }

Mikhail Belikov

@mbelikov_twitter

hi guys! where can I find a list of prominent projects using the parboiled2?

just need some production ready examples of parboiled2 usage

Alexander Myltsev

@alexander-myltsev

@umbreak what’s the question?

@mbelikov_twitter a good point to start is to look at usages at MVN Repository:
https://mvnrepository.com/artifact/org.parboiled/parboiled/usages

Boris V.Kuznetsov

@tampler

@mbelikov_twitter I have several prominent projects, which use Parboiled2. But they are all commercial, so I can't share. You are welcome to ask concrete questions

Boris V.Kuznetsov

@tampler

Hello! I implemented some rule def Node: Rule[String :: String :: HNil, NodeAST :: HNil], which compiles fine. However, when I try to add it to the root rule, I'm getting an error message: Theoptional,zeroOrMore,oneOrMoreandtimesmodifiers can only be used on rules of typeRule0,Rule1[T]andRule[I, O <: I]!
How do I fix that ? Thanks

I'm adding it as oneOfMore(Node)

Boris V.Kuznetsov

@tampler

Okay, fixed by changing the rule to def Node: Rule[HNil, NodeAST :: HNil]. Pls disregard this question. Thank you for a great library !

Boris V.Kuznetsov

@tampler

Hello ! I'm thinking about the best approach to attack the following parsing problem. Suppose I have the following input grammar:

    when _T : 
      tmp <= UInt<1>("h00") 
      skip 
    else : 
      node _T_1 = add(tmp, UInt<1>("h01")) 
      node _T_2 = tail(_T_1, 1) 
      skip

That's essentially a logic block, which spans multiple lines. It starts with a when statement and has two branches : when and else. As far as the number of lines in each branch is variable (zero to many) and right-hand-side deps may be implemented aside the when block, I can not put this statement into a fixed case class like

final case class MyWhen( cond:String, node0: Node, node1:Node)

How would you parse this input? Any advice and example is appreciated . Thank you!

Boris V.Kuznetsov

@tampler

Is there any way to selectively parse the input? Say, I'd love to have a single pass, which parses only lines, which start with input or output like this:

figure Area :
  topic Essential:
    input vision : Vision
    output sun : Sun

Boris V.Kuznetsov

@tampler

@alexander-myltsev any thoughts ?

Alexander Myltsev

@alexander-myltsev

@tampler

How would you parse this input? Any advice and example is appreciated

I’m not following what the problem is. derive Expr(statement: Seq[?]) from the Node.

final case class MyWhen( cond:String, thenExpr: Node, elseExpr:Node)

@tampler

Is there any way to selectively parse the input? Say, I'd love to have a single pass, which parses only lines, which start with input or output like this:

I suggest you to extract those lines with RegEx. and then apply a parser

Behrang

@behrangsa

Hi all,

When defining a grammar in Parboiled 1, is there an option to specify that a specific term/token should only appear {n,m} times?

Also is it possible to create a Parboiled parser for a data format language like this:

SIZE CHAR{SIZE}

where CHAR appears SIZE times AND SIZE is a two digit zero padded number, e.g:

05HELLO
11PROGRAMMING

1 reply

Alexander Myltsev

@alexander-myltsev

@behrangsa This is a parboiled2 group. I'm afraid there are not many people who know parboiled1.

Behrang

@behrangsa

Unfortunately there's no one in my body who knows Scala :(

Boris V.Kuznetsov

@tampler

I suggest you to switch to Parboiled2 then. No reasons to use an old barely alive Parb1

Tobias Haf

@tobias.haf_gitlab

Hi, I used Parboiled1 (Java) in the past and I'm currently evaluating Parboiled2 due to its performance promises :-) I'm wondering how to provide proper location information (startOffset, endOffset) to the AST nodes pushed to the value stack.
In Parboiled1 I had two Actions within a rule; first ACTION to push the node with the current offset (startOffset) to the value stack and second ACTION to set the endOffset to this node:

Sequence(
    ACTION(factory.createNode()),
    matchingRules(),
    ACTION(factory.finalizeNode())
)

How should this be implemented in Parboiled2?

2 replies

Where communities thrive

People

Repo info

Activity