Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    bdanielby
    @bdanielby
    Try this:
    def anyFollowedByMinus[_: P]: P[String] = P((CharsWhile(c => c != '-', 0) ~ "-").rep.!)
    bdanielby
    @bdanielby
    @sake92 What about this?
      def closingParen[_: P]: P[Option[String]] = P(")".!).map(s => Some(s))
    Waleed Khan
    @arxanas
    Hi, is there a way to include the positions of the matched nodes in the output? I see that result of parsing will include the index at which the parsing completed/failed, but what if I want to know the positions of the successfully-parsed nodes?
    Siddhant Sanyam
    @siddhant3s

    Hi @lihaoyi, thanks a lot for writing this excellent blog post on building programming language. I found this post and one on Parser combinator very useful. At the end, I noticed you recommended reading "Crafting Interpreters". I think this book is excellent and a very gentle introduction to writing a programming language.

    I finished Crafting Interpreters and I was wondering if you have any advice/pointers on similar resource but for writing statically typed language. I want to add type-checking to Lox. Is there particular resource you have in mind which might be useful if one needs to add static type-checking. So far most of the papers I come across are quite focused on functional languages: "Types and Programming Language by Pierce", "Implementation of FP languages by Simon Peter Jones" etc. Although they are probably good resources, I was trying to see if there is anything that exist as easy going as "Crafting Interpreters" is. The closest was "Write you a Haskell" but it isn't complete.

    Thank you :)

    TimmyC101
    @TimmyC101
    Good afternoon, all. I'm working on developing a parser for an iCalendar library, and wanted to verify that fastparse was fit for purpose before learning the ins and outs of the lib. Specifically, I need to be able to parse strings in any order, and a variable number of times (0, once, multiple). Is fastparse sufficient for this, or is it fairly rigid in terms of ordering and frequency? If sufficient, directing me toward a similar use case would be much appreciated! Thank you.
    Ichoran
    @Ichoran
    Sure, if you can't handle that you can't parse much. Use rep and | a lot. However, if the strings aren't easily distinguished from each other, the backtracking to try alternative branches might be kind of slow.
    If the strings are easily separated from each other, and there's a quick way to distinguish who is whom without needing to do a full parse, you might want to split the strings apart some other way and send each string to a parser for its type. You don't necessarily need to, but for line-by-line parses I generally find that simpler conceptually than making one parser for a whole file.
    TimmyC101
    @TimmyC101
    Awesome, thanks for the reply!
    Waleed Khan
    @arxanas
    @siddhant3s it will probably depend on what kind of static type system you want to add. If you want to add a bidirectional type system, it might be easier, because the type inference/checking process will be similar to AST traversals. A Hindley-Milner system is not overly complicated either, but you have to do a unification step afterwards, so it's a little bit harder to understand intuitively. There are a lot of tutorials for Hindley-Milner (maybe try https://www.youtube.com/watch?v=cQf_6NsLp80 ?), and here's one for bidirectional type inference: http://davidchristiansen.dk/tutorials/bidirectional.pdf (not "easy-going", but the easiest I could find)
    Waleed Khan
    @arxanas
    Is there a strategy for error-recovery with parser combinators? For example, if I want to fail parsing of the current statement/expression and proceed to the next synchronization token. Do I have to just annotate every parse which might fail with a | and something that consumes all the input until the next synchronization token?
    Ichoran
    @Ichoran
    That's how I generally do it.
    Srinivasa Reddy
    @psrinuhp_twitter
    I have started learning fastparse.. i have input "/user/@name/location/@age/old/". could any one please help me to extract only @name,@age fields.
    nafg
    @nafg
    @psrinuhp_twitter for something that simple I would just use a regex
    Srinivasa Reddy
    @psrinuhp_twitter
    Sure, Thank you.
    Joerg von Roos
    @jvr-ks

    // Tryout in SBT REPL, i.e. sbt -consoleQuick

    import fastparse._, NoWhitespace._
    import scala.collection.immutable.TreeSet
    
    
    val input = "/user/Mr. Smith/location/49/old/"

    // 1. start with:
    def parsePattern[: P] = P("/user/" ~ AnyChar.rep(1).!)
    val Parsed.Success(resultValue,
    ) = parse(input, parsePattern(_))
    // val resultValue: String = Mr. Smith/location/49/old/

    // 2. then try:
    def parsePattern[: P] = P("/user/" ~ (!"/" ~ AnyChar).rep(1).!)
    val Parsed.Success(resultValue,
    ) = parse(input, parsePattern(_))
    // val resultValue: String = Mr. Smith

    // 3. then try:
    def parsePattern[: P] = P("/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/")
    val Parsed.Success(resultValue,
    ) = parse(input, parsePattern(_))
    // val resultValue: (String, String) = (Mr. Smith,49)

    // 4. then try:
    val input = """/user/Mr. Smith/location/49/old/
    /user/Mr. Miller/location/23/old/
    """
    def parsePattern[: P] = P("/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/")
    def parsePatternMulti[
    : P] = P(parsePattern.rep(1))
    val Parsed.Success(resultValue, ) = parse(input, parsePatternMulti())
    // val resultValue: Seq[(String, String)] = List((Mr. Smith,49))

    // 5. then try:
    val input = """/user/Mr. Smith/location/49/old/
    /user/Mr. Miller/location/23/old/
    """
    def parsePattern[: P] = P("/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/" ~ (!"/" ~ AnyChar).rep(0))
    def parsePatternMulti[
    : P] = P(parsePattern.rep(1))
    val Parsed.Success(resultValue, ) = parse(input, parsePatternMulti())
    // val resultValue: Seq[(String, String)] = List((Mr. Smith,49), (Mr. Miller,23))

    // 6. then try:
    val input = """/user/Mr. Smith/location/49/old/
    /user/Mr. Miller/location/23/old/
    """
    def parsePattern[: P] = P(Index ~ "/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/" ~ (!"/" ~ AnyChar).rep(0))
    def parsePatternMulti[
    : P] = P(parsePattern.rep(1))
    val Parsed.Success(resultValue, ) = parse(input, parsePatternMulti())
    // val resultValue: Seq[(Int, String, String)] = List((0,Mr. Smith,49), (33,Mr. Miller,23))

    import fastparse._, NoWhitespace._
    import scala.collection.immutable.TreeSet
    
    
    val input = "/user/Mr. Smith/location/49/old/"
    
    
    // 1. start with:
    def parsePattern[_: P] = P("/user/" ~ AnyChar.rep(1).!)
    val Parsed.Success(resultValue, _) = parse(input, parsePattern(_))
    // val resultValue: String = Mr. Smith/location/49/old/
    
    
    // 2. then try:
    def parsePattern[_: P] = P("/user/" ~ (!"/" ~ AnyChar).rep(1).!)
    val Parsed.Success(resultValue, _) = parse(input, parsePattern(_))
    // val resultValue: String = Mr. Smith
    
    // 3. then try:
    def parsePattern[_: P] = P("/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/")
    val Parsed.Success(resultValue, _) = parse(input, parsePattern(_))
    // val resultValue: (String, String) = (Mr. Smith,49)
    
    // 4. then try:
    val input = """/user/Mr. Smith/location/49/old/
    /user/Mr. Miller/location/23/old/
    """
    def parsePattern[_: P] = P("/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/")
    def parsePatternMulti[_: P] = P(parsePattern.rep(1))
    val Parsed.Success(resultValue, _) = parse(input, parsePatternMulti(_))
    // val resultValue: Seq[(String, String)] = List((Mr. Smith,49))
    
    
    // 5. then try:
    val input = """/user/Mr. Smith/location/49/old/
    /user/Mr. Miller/location/23/old/
    """
    def parsePattern[_: P] = P("/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/" ~ (!"/" ~ AnyChar).rep(0))
    def parsePatternMulti[_: P] = P(parsePattern.rep(1))
    val Parsed.Success(resultValue, _) = parse(input, parsePatternMulti(_))
    // val resultValue: Seq[(String, String)] = List((Mr. Smith,49), (Mr. Miller,23))
    
    // 6. then try:
    val input = """/user/Mr. Smith/location/49/old/
    /user/Mr. Miller/location/23/old/
    """
    def parsePattern[_: P] = P(Index ~ "/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/" ~ (!"/" ~ AnyChar).rep(0))
    def parsePatternMulti[_: P] = P(parsePattern.rep(1))
    val Parsed.Success(resultValue, _) = parse(input, parsePatternMulti(_))
    // val resultValue: Seq[(Int, String, String)] = List((0,Mr. Smith,49), (33,Mr. Miller,23))
    TreeSet not used ...
    Joerg von Roos
    @jvr-ks
    Only a starting point...
    Joerg von Roos
    @jvr-ks
    // 7. using a TreeSet to rmove duplicates and sort:
    
    import fastparse._, NoWhitespace._
    import scala.collection.immutable.TreeSet
    
    val input = """/user/Mr. Smith/location/49/old/
    /user/Mr. Miller/location/23/old/
    """
    
    class Data(val index: Int, val name: String, val age: String) extends Ordered[Data] {
        def compare(that: Data): Int = {
            if (this.index == that.index){
                0
            } else {
                if (this.index > that.index){
                    1
                } else {
                    -1
                }
            }
        }
    }
    
    val dataSet = TreeSet.newBuilder[Data]
    
    def parsePattern[_: P] = P(Index ~ "/user/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/location/" ~ (!"/" ~ AnyChar).rep(1).! ~ "/old/" ~ (!"/" ~ AnyChar).rep(0))
    def parsePatternMulti[_: P] = P(parsePattern.rep(1))
    
    parse(input, parsePatternMulti(_)) match {
      case Parsed.Success(resultValue, _) => {
    
        for (i <- 0 until resultValue.length){
          dataSet += new Data(resultValue(i)._1, resultValue(i)._2, resultValue(i)._3)
        }
      }
    
      case f @ Parsed.Failure(_, index, _) => {
        val trace = f.trace()
        val msgLong = trace.longAggregateMsg
        val erLong = s"Parse error: $msgLong (index: $index)"
        println(erLong)
      }
      case _ => println("Severe parse error!")
    }
    
    val r = dataSet.result()
    
    r foreach {x =>
      println(s"index: ${x.index} name: ${x.name} age: ${x.age}")
    }
    Hanns Holger Rutz
    @Sciss
    Hi there. Looking into options for replacing scalariform for lexing Scala source code, especially supporting both Scala 2.12 and 3.0 syntax. I understand that ScalaParse is an example for Scala 2 shipped with fastparse. Is Scala 3 grammar in the cards?

    Also the doc says

    ScalaParse does not currently generate an AST

    Does it mean I wouldn't be able to get a token stream, or is that something I can get from ScalaParse? I would want to use it for syntax highlighting.

    Li Haoyi
    @lihaoyi
    Ammonite uses scalaparse for syntax highlighting without generating a token stream; you can look at the code and see how it's done with intercept and tracking character ranges
    Hanns Holger Rutz
    @Sciss
    Ok thanks. So I guess I could populate a token stream then as a side effect of the interception
    bdanielby
    @bdanielby
    But fastparse won't be able to work on a token stream in a second run. It can only parse strings or character streams. One usually produces an abstract syntax tree instead.
    bdanielby
    @bdanielby
    Of course if all you need is a lexer then that's okay.
    Hanns Holger Rutz
    @Sciss
    yes, for now just a lexer
    ㄗㄠˋ ㄑㄧˊ
    @tsao-chi
    I want to convert P[Option[T]] to P[T]. Are there functions to construct P[T] that always fail or always produce a certain T?
    Li Haoyi
    @lihaoyi
    .map(_.get)
    ?
    or .flatMap and send the None case to Fail
    ㄗㄠˋ ㄑㄧˊ
    @tsao-chi
    Why is failure not a case class?
    Li Haoyi
    @lihaoyi
    because it has some funniness around keeping a lambda around for re-running the parse with tracing, so it isn't really "pure" data
    Luis Angel Vicente Sanchez
    @lvicentesanchez
    Has anyone tried to build polish notation calculator using fastparse?
    I tried to tweak the infix calculator in this post: https://kubuszok.com/2019/from-string-to-ast-parsing/
    The code is this one
    import fastparse._
    import SingleLineWhitespace._
    
    object AST {
      sealed trait Terminal extends Product with Serializable
    
      final case class Number(value: java.lang.Number) extends Terminal
    
      sealed trait BinaryOperator
      case object Plus  extends BinaryOperator with Terminal
      case object Minus extends BinaryOperator with Terminal
      case object Times extends BinaryOperator with Terminal
      case object Div   extends BinaryOperator with Terminal
    
      sealed trait Expression extends Product with Serializable
      final case class FromNumber(number: Number)                                                  extends Expression
      final case class FromBinary(operand1: Expression, operand2: Expression, bin: BinaryOperator) extends Expression
    }
    
    object Parser {
      import AST._
    
      // terminals
      def number[_: P] =
        P(CharIn("0-9").rep(1).!).map(n => Number(n.toInt))
      // ! makes parser catch input as String
      def plus[_: P]  = P("+").map(_ => Plus)
      def minus[_: P] = P("-").map(_ => Minus)
      def times[_: P] = P("*").map(_ => Times)
      def div[_: P]   = P("/").map(_ => Div)
    
      // non-terminals
      def binaryOperator[_: P] = P(plus | minus | times | div)
      def fromNumber[_: P]: P[FromNumber] =
        P(number.map(FromNumber(_)))
      def fromBinary[_: P]: P[FromBinary] =
        P(
          (binaryOperator ~ (fromNumber | inParenthesis) ~ (fromNumber | inParenthesis)).map {
            case (op, ex1, ex2) => FromBinary(ex1, ex2, op)
          }
        )
      def expression[_: P] =
        P(fromBinary | fromNumber)
      def inParenthesis[_: P] =
        P("(" ~ expression ~ ")")
    
      def program[_: P] = P((expression | inParenthesis) ~ End)
    }
    And parse("+ 12 23", Parser.program(_)) emits this error java.lang.NumberFormatException: For input string: "12 23"
    If I change the number parser to P(CharIn("0-9").rep(1, sep = " ").!).map(n => Number(n.toInt)), then I get this error Parsed.Failure(Position 1:6, found "23"): fastparse.Parsed[AST.Expression]
    Luis Angel Vicente Sanchez
    @lvicentesanchez
    I know the problem is the handling of spaces, but I'm not sure how to continue without writing my own whitespace handler, and that seems a bit too complicated for something as simple as this
    Luis Angel Vicente Sanchez
    @lvicentesanchez
    and the solution is to use repX for the number
      def number[_: P] =
        P(CharIn("0-9").repX(1).!).map(n => Number(n.toInt))
    and now the output is Parsed.Success(FromBinary(FromNumber(Number(12)),FromNumber(Number(23)),Plus), 7)
    JoRo
    @fluxlife

    Hi all, I am new to fastparse and am having an issue. I have a function like this: def parseMyStuff(lines: String): Option[Seq[MyCaseClass]] and inside the function I build up my parser and then call the parse(lines, myParser(_) match … etc. Whenever I try to call that function from anywhere in my app I am getting a No implicits found for parameter P … error.

    What am I doing wrong?

    JoRo
    @fluxlife
    I think I found the solution to my problem by mimic’ing the Json object in this test: https://github.com/com-lihaoyi/fastparse/blob/master/fastparse/test/src/fastparse/JsonTests.scala
    Dan Hodges
    @DanHodges

    Hey folks,

    Can I get a workflow approval/review at your convenience?
    com-lihaoyi/fastparse#256

    Thanks :)

    Lorenzo Gabriele
    @lolgab
    Hi @DanHodges,
    I can review it :)
    Lorenzo Gabriele
    @lolgab
    @DanHodges Left a comment on the PR :)
    Pyry-Samuli Lahti
    @Pyppe
    Hi! I'm prototyping with a simple boolean parser, but having a problem: https://scastie.scala-lang.org/Pyppe/URhgdSiBQnCVg9fjlU7rAA Any ideas why e.g. simple 1 OR 2 is not working without the parenthesis?
    Daniel Mateus Pires
    @dmateusp
    Hey do I have a way to say "match this parser as many times as possible and anywhere in the given input" ?
    micky44
    @Micky44Scoll_twitter
    Hi guys, have scala fastparse support scala 3?