Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 19:08

    travisbrown on master

    Document how to update the docu… (compare)

  • 19:08
    travisbrown closed #1558
  • 19:08
    travisbrown commented #1558
  • 14:51
    x3ro commented #1552
  • 14:49
    x3ro review_requested #1558
  • 14:49
    x3ro opened #1558
  • 14:29

    travisbrown on master

    Link directly to `io.circe` pac… (compare)

  • 14:29
    travisbrown closed #1552
  • 14:28
    travisbrown commented #1552
  • 13:22
    x3ro commented #1552
  • Oct 20 15:23
    feoktant commented #726
  • Oct 20 08:41
    travisbrown closed #1529
  • Oct 20 08:39

    travisbrown on master

    Update refined, refined-scalach… (compare)

  • Oct 20 08:39
    travisbrown closed #1541
  • Oct 20 08:35

    travisbrown on master

    Update sbt-scalajs, scalajs-com… (compare)

  • Oct 20 08:35
    travisbrown closed #1550
  • Oct 20 08:33
    travisbrown closed #1556
  • Oct 20 08:33

    travisbrown on master

    Update sbt-mima-plugin to 0.8.1… (compare)

  • Oct 20 08:33

    travisbrown on master

    Update sbt-dotty to 0.4.4 (#155… (compare)

  • Oct 20 08:33
    travisbrown closed #1555
Lucas Satabin
@satabin
@covid82 would something like that work for you: https://scastie.scala-lang.org/I0ReqHcvRU2R2sW258bRnw ?
covid82
@covid82

@satabin Thanks for that! This is exactly what I was trying to achieve!
Here is the version (slightly changed) that I used:

package json.parser

import java.nio.file.Paths
import java.util.concurrent.Executors

import cats.effect.{Blocker, ExitCode, IO, IOApp}
import cats.implicits._
import io.circe._
import fs2._
import fs2.data.json._
import fs2.data.json.circe._

import scala.util.Try

object MyFs2DataStreamingParser extends IOApp {
  def run(args: List[String]): IO[ExitCode] = {
    val selector = ".order.items.[]".parseSelector[Try].get

    def nullify(json: Json): Json =
      json.hcursor.downField("quantity").as[Int]
        .map(qty => if (qty > 1) Json.Null else json)
        .fold(_ => Json.Null, identity)

    val file = Paths.get("test1.json")
    Blocker.fromExecutorService[IO](IO(Executors.newCachedThreadPool())).use { blocker =>
      val stream = fs2.io.file.readAll[IO](file, blocker, 8192)
        .through(fs2.text.utf8Decode[IO])
        .flatMap(chunk => Stream.fromIterator[IO](chunk.toCharArray.iterator))
        .through(tokens)
        .through(transform[IO, Json](selector, nullify))
        .filter(_ != Token.NullValue)
        .compile
        .to(collector.pretty())
        .flatMap(x => IO(println(x)))
      for {
        start <- IO(System.currentTimeMillis())
        _ <- stream
        _ <- IO(println(s"completed in ${(System.currentTimeMillis() - start).toDouble / 1000.toDouble}s"))
      } yield ()
    }.as(ExitCode.Success)
  }
}

I did a quick performance test:

   10000 elements - completed in 0.89s
  100000 elements - completed in 8.852s
 1000000 elements - completed in 67.74s
10000000 elements - completed in 729.227s

Is it a good practice to use cachedThreadPool as a blocker? I noticed that if I use blocker from Blocker[IO].use { blocker => ...} it is not very performant.
Is there are a better way to get stream of characters from file - I am not sure this if the approach that I came up with is a recommended practice - flatMap(chunk => Stream.fromIterator[IO](chunk.toCharArray.iterator)).
Again, thank you all for your help!

Lucas Satabin
@satabin
I usually use .flatMap(Stream.emits(_)) to have a stream of Char out of a stream of Strings
just realized that the .fold(_ => Json.Null, identity) is probably better as .getOrElse(Json.Null)
covid82
@covid82
Cool! I'll try that! Thank you for the great libraries by the way! You all are doing amazing work! :)
Andrew Ehrlich
@ehrlichja
good morning. is there any feature in Circe for querying by json path? I had thought that .downField might accept a deeply nested path, like "outer.inner1.inner2", but I guess not.
Gavin Bisesi
@Daenyth
There's optics-based JsonPath stuff, check the docs iirc
Andrew Ehrlich
@ehrlichja
yea. but I was hoping for something that can parse a string instead of using Dynamic
Travis Brown
@travisbrown
@ehrlichja I think I remember somebody doing something like that, but I can't remember where. Whenever I've needed something like that myself, it's always been fairly simple, and I've just parsed the string via split, etc. into a cursor operation.
Andrew Ehrlich
@ehrlichja
i was thinking about that too
im receiving an infinite stream of K/V where the key is a json path and the value is the value to set
but the records can be very deeply nested, and contain array indices, and so on
Travis Brown
@travisbrown
How arbitrary are the paths?
Andrew Ehrlich
@ehrlichja
very arbitrary
Travis Brown
@travisbrown
hmm, yeah, array indices are more complicated
Lucas Satabin
@satabin
@ehrlichja are they Json Pointers as in RFC-6901?
Andrew Ehrlich
@ehrlichja
the seperators are dots but other than that looks similar
Lucas Satabin
@satabin
ok, then diffson won't help you here, you'll probably need to parse it yourself :/
Andrew Ehrlich
@ehrlichja
i can sort of envision splitting on the dots and then folding, either applying downField or downN on each element
Lucas Satabin
@satabin
this would be my approach as well
xmoooz
@xmoooz
why for using akka-http with circe i must use :
https://github.com/hseeberger/akka-http-json
Travis Brown
@travisbrown
@xmoooz You don't have to, it just lets you avoid some boring connecting code.
xmoooz
@xmoooz
@travisbrown would you explain more or give me a link. thanks
Tushar Mathur
@tusharmath
Hi, when should I use semi auto vs auto? I am always confused.
Paul Roman
@romusz

Hi, Has anybody got codec derivation working with Dotty. I tried Circe 0.13.0 with Dotty 0.22.0-RC1 as per info in the release notes - tried the example, didn't get far:

scala> import io.circe.Codec
     | 
     | sealed trait Event derives Codec.AsObject
3 |sealed trait Event derives Codec.AsObject
  |                                 ^
  |no implicit argument of type deriving.Mirror.Of[Event] @InlineParam was found for parameter A of method derived in trait CodecDerivation

Any info appreciated.

Tushar Mathur
@tusharmath
Is there a better way to reduce this boilerplate —
  implicit def decoder: Decoder[Client] =
      List[Decoder[Client]](
        decoderGqlStart.prepare(_.downField("GQL_START")).widen,
        deriveDecoder[GQL_CONNECTION_INIT].prepare(_.downField("GQL_CONNECTION_INIT")).widen,
        deriveDecoder[GQL_STOP].prepare(_.downField("GQL_STOP")).widen,
        deriveDecoder[GQL_CONNECTION_TERMINATE].prepare(_.downField("GQL_CONNECTION_TERMINATE")).widen
      ).reduceLeft(_ or _)
Zeshan Ali
@zeshansali

Hey all, I’m trying to spin up a small API using http4s and circe. I’m running into an issue when trying to encode a simple case class using automatic derivation. The code is:

import java.{util => ju}

import cats._
import cats.effect._
import cats.implicits._
import doobie._
import doobie.free.Embedded.Connection
import doobie.implicits._
import doobie.postgres._
import doobie.postgres.implicits._
import doobie.util.ExecutionContexts
import io.circe._
import io.circe.generic.auto._
import io.circe.literal._
import io.circe.syntax._
import org.http4s._
import org.http4s.dsl.io._
import org.http4s.implicits._
import org.http4s.server.blaze._
import org.http4s.server.Router

case class User(firstName: String,
                lastName: String,
                birthday: ju.Date,
                email: String,
                createdAt: String,
                updatedAt: String)

trait UserRepo {
  def getUser(id: String): IO[User]
}

case class UserRepoImpl(xa: Transactor[IO]) extends UserRepo {
  override def getUser(id: String): IO[User] = {
    sql"""
          |select first_name,
          |       last_name,
          |       birthday,
          |       email,
          |       created_at,
          |       updated_at
          |  from users
          | where id = ${ju.UUID.fromString(id)}
          |
       """.stripMargin
          .query[User]
          .unique
          .transact(xa)
  }      
}

object UserRoutes {
  def get(userRepo: UserRepo): HttpRoutes[IO] = {
    HttpRoutes.of[IO] {
      case GET  -> Root / id => {
        userRepo.getUser(id).flatMap(user => Ok(user.asJson))
      }
    }
  }
}

object Main extends IOApp {
  implicit val cs = IO.contextShift(ExecutionContexts.synchronous)

  val xa = Transactor.fromDriverManager[IO]("org.postgresql.Driver", "jdbc:postgresql:local", "postgres", "password")
  val userRepo: UserRepo = UserRepoImpl(xa)

  val httpApp = Router(
    "/api/v1/users" -> UserRoutes.get(userRepo)
  ).orNotFound

  def run(args: List[String]): IO[ExitCode] =
    BlazeServerBuilder[IO]
      .bindHttp(8080, "localhost")
      .withHttpApp(httpApp)
      .serve
      .compile
      .drain
      .as(ExitCode.Success)
}

I get the following error when I try to compile (via sbt):

could not find implicit value for parameter encoder: io.circe.Encoder[com.zsali.User]
         userRepo.getUser(id).flatMap(user => Ok(user.asJson))

I think I’m doing everything the docs mention for automatic derivation, but can’t tell if I’m missing something. Can anyone help me understand what I’m doing wrong? I’m using circe 0.13.0 and Scala 2.13.2. Thanks!

ybasket
@ybasket
@zeshansali circe doesn’t come with an Encoder for java.util.Date, so define your own or use java.time.LocalDate
ComFreek
@ComFreek
I am running into a stack overflow with the Scala compiler when compiling some code with nested case classes and io.circe.generic.auto._.
I tried increasing the stack size -- as mentioned in the docs: https://circe.github.io/circe/codecs/known-issues.html -- to no avail.
To be precise, I tried increasing it in IntelliJ's settings pane (the IntelliJ project is imported from an SBT build file). I am not entirely sure that IntelliJ picks that up.
ComFreek
@ComFreek
I now tried the native SBT shell. Same error: https://pastebin.com/frNLSVhj
I had set the stack size via javaOptions in Global ++= Seq("-Xmx2g", "-Xss512m") in my build.sbt.
ComFreek
@ComFreek
Is there any way to circumvent the problem?
I have now tried to provide explicit implicits implicit val enc = Encoder[MyCaseClass], implicit val dec = Encoder[MyCaseClass]. It works in IntelliJ (only by coincidence?), but not from the SBT shell.
Zeshan Ali
@zeshansali
@ybasket That worked - thanks for the help.
Felix Palludan Hargreaves
@hejfelix
Can I compose multiple Plated transformations such that I don't have to traverse the structure multiple times?
Alex Antonov
@antonoal
Hi. I've just updated the libs versions on my project and got java.lang.NoSuchMethodError: fs2.Pull$.stream$extension(Lfs2/internal/FreeC;)Lfs2/internal/FreeC;. I believe it's cause by the fact that circe-fs2 v 0.13 is built against fs2 v2.2.2. Is there a release planned built for fs2 v 2.4.x any time soon or should I hold on with the upgrade?
Gavin Bisesi
@Daenyth
Sounds more like an error in fs2 that 2.4 breaks bincompat with 2.2
:eyes: yes fs2 2.4 says it's binary compatible with previous 2.x releases
Damian van Beekum
@theejazz
Hi! I've been having trouble trying out automated decoding. I keep getting a DecodingFailure, specifically "Attempt to decode value on failed cursor, List(DownField(hits))" where 'hits' is a key in my JSON. Interestingly, the key it prints often appears to be the very last item in my JSON (It's not always, but that might be a caching issue). Any suggestions as to how I might fix this? Below is a simple example that produces this result for me:
import io.circe.generic.auto._
import io.circe.parser.decode

case class Request(meta: Meta, response: Response)
case class Meta(status: Int, message: Option[String])
case class Response(hits: String)

object RequestDecoder {
  def main(args: Array[String]): Unit = {
    val request = """{"meta":{"status":200},"response":{"hits":"hi"}}"""
    println(decode[Response](request))
  }
}
Yilin Wei
@yilinwei
@theejazz I'm on a phone so the formatting may be off, but isn't that a Request rather than a Response?
Alex Antonov
@antonoal
I'll raise it with fs2, thanks @Daenyth
Alex Antonov
@antonoal
Actually it's my mistake - no issue with fs2
Damian van Beekum
@theejazz
@yilinwei wow I'm genuinely facepalming right now... Thanks!
Ashwin Bhaskar
@ashwinbhaskar

I start getting this error after adding http4s-circe dependency.

 The json interpolator requires jawn to be available at compile time
[error]         val expectedJson = json"""

Why does this happen and how can i resolve it?

Gavin Bisesi
@Daenyth
Add jawn at % Compile scope
Tharun Kumar Kapse
@tkapse
Hi All, I am trying to decode JsonArray with ~900 objects where it complains with malformed json error. When I reduce the json count to ~110 objects it works properly. I am trying to understand is there any limit on no. of object in an array or is it the data size that's causing this issue?
cld u please help me out
Ashwin Bhaskar
@ashwinbhaskar
@Daenyth but it worked before I introduced http-circedependency.
Gavin Bisesi
@Daenyth
weird. I don't know