Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Fred Jonsson
    @enginoid
    Right, makes sense. I guess since this error occurs when we're mapping over rows, I might as well print the rows as we encounter them.
    Jeffrey N. Davis
    @penland365
    also, check what columns are being returned - you can do that via
    result.columns.foreach(c => println(c.name))
    The postgres style is to send a specific error code ( there are hundreds ) if anything at all goes wrong during qury phase. The fact that the result returns correctly and you are able to map over the result at all suggests that, from Postgres perspective, the Client Request / Response performed correctly
    Fred Jonsson
    @enginoid

    Thanks for your help @penland365. I've added logging and just attempted to reproduce the problem. I wasn't able to reproduce it right now (even though I've been able to pretty consistently), perhaps because:

    • The servers are in some kind of "hot state" after the deployment and they need to stand idling for a little bit before we can reproduce. (I'm skeptical of this explanation, at least in JIT terms they should be cold as ever since we're not pre-warming this server.)
    • The players I've requested are in postgres cache, and fast retrieval is less likely to trigger the race.
    • Logging is interfering with the race.

    At any rate, I'm going to leave the logging on debug overnight and wait for this to re-occur. Thanks for your help -- I look forward to reporting back!

    Jeffrey N. Davis
    @penland365
    Thanks!
    Fred Jonsson
    @enginoid
    Unfortunately, what I have to report at the moment is that the bug seems to be entirely gone. The last occurrence of the exception is right before I deployed the change. The system is used around the clock and we were getting a few errors every hour, so we should definitely have an error at this point (16 hours later).
    What I'm looking into now is whether the roc dependency changed and whether this was potentially a bug that is now fixed.
    Fred Jonsson
    @enginoid
    It looks like no significant dependencies or shared packages changed between the two revisions.
    Fred Jonsson
    @enginoid
    I only implemented logging in one endpoint, but multiple endpoints were failing with similar errors, so logging does not seem the cause.
    Out of insanity, I'm going to try turning logging off and see what happens.
    Jeffrey N. Davis
    @penland365
    :(
    Fred Jonsson
    @enginoid
    Logging has actually helped re-surface the errors (if there's at all a causal relationship), so I'm about to turn it back on.

    I've also discovered an interesting encoding issue that I'm trying to track down at the moment. I thought it would come down to configuration, but at this stage there's not a lot to configure in the client. I set up these functions to execute against a local postgres instance that I spun up:

    def getFood(): Future[String] = {
      val req = new Request(s"SELECT 'Empeñada' as course".stripMargin)
      pgClient.query(req).map { result =>
        result.map(_.get('course).as[String]).head
      }
    }
    
    def show(symbol: Symbol): Future[Seq[String]] = {
      val req = new Request(s"show ${symbol.name}".stripMargin)
      pgClient.query(req).map { result =>
        result.columns.foreach(column => println(column.name))
        result.map(_.get(symbol).as[String]).toSeq
      }
    }

    When I call getFood, it looks like something has gone wrong in the decoding of the string:

    scala> getFood().get
    res5: String = Empeᅢᄆada

    The client encoding and server encoding both seem set to UTF-8:

    scala> show('server_encoding).get
    res6: Seq[String] = List(UTF8)
    
    scala> show('client_encoding).get
    res7: Seq[String] = List(UTF8)
    At first I thought textDecoder was at fault, so I added a unit test with unicode characters but that one passed with flying colors
    Jeffrey N. Davis
    @penland365
    On the getFood().get comment, what terminal are you using?
    There is certainly a possibility the encoders are wrong, but they are tested w/ ScalaCheck, which generates a preponderance of Special Unicode characters during testing
    Fred Jonsson
    @enginoid
    Apple's Terminal with default settings. Let's see if I can tease the actual unicode values out of this with some Scala function to make sure that's not it.
    Fred Jonsson
    @enginoid
    I'm sure there are better ways to get at this, but it does look like this is the actual encoding of the string:
    scala> val food = getFood().get
    food: String = Empeᅢᄆada
    
    scala> food.foreach(char => println("%02X" format char.hashCode))
    45
    6D
    70
    65
    FFC3
    FFB1
    61
    64
    61
    I'll file away an issue for now. Please let me know if you have a hunch as to what this might be -- I might be able to dig in.
    Jeffrey N. Davis
    @penland365
    I think I see the error
    That needs to be
    val strValue = new String(bytes, StandardCharsets.UTF_8)
     Text(column.name, column.columnType, strValue)
    I hadn’t written tests yet for much of the Result, since that API was going to change at some point
    @enginoid Thanks for catching this!
    Can you log a bug for me on github? I’ll push a change today
    Fred Jonsson
    @enginoid
    Ahh, I see. Thanks for looking into it!
    Will do!
    Fred Jonsson
    @enginoid
    I've filed this away here: finagle/roc#70
    Jeffrey N. Davis
    @penland365
    Thanks so much @enginoid
    jeremyrsmith
    @jeremyrsmith
    Hi there, am I right in thinking that roc's Binary is used for the postgres binary protocol?
    was poking around the source and some of the binary decoders seem to assume a text format (i.e. the timestamp one)
    I have many of the decoders for postgres binary values implemented in scala if that would be useful
    Jeffrey N. Davis
    @penland365
    @jeremyrsmith Hi! Sorry, I’ve been on vacation recently.
    Yes, you’re 100% correct, Binary is meant to represent the Binary Postgresql encodings.
    If you’ve got many of the binary decoders that would be a HUGE help for the project - I absolutely could be wrong on some of the format, I got some of the binary encodings by looking at the Postgres jdbc code
    jeremyrsmith
    @jeremyrsmith
    Can CircleCI provide a postgres server during test like Travis can?
    I've found a good way to test binary decoders is to use a postgres server and push ScalaCheck generators through it like SELECT int4send($1::int4);
    jeremyrsmith
    @jeremyrsmith
    I noticed in some spots you use NIO ByteBuffer to wrap the byte array and read it, and in others you make a new netty buffer and copy the byte array into it; is one preferred over the other?
    there's also a bit of an issue with ambiguity - sometimes a logical scala type will map to two different type OIDs that have different codecs. IE Json - you would probably think use as[Json] for both json and jsonb but these have slightly different codecs and currently the ElementDecoder[Json] instance won't know which to use since it doesn't receive the OID.
    jeremyrsmith
    @jeremyrsmith
    @penland365 finagle/roc#72
    what's the plan for parameterized queries? Will there be an ElementEncoder typeclass for passing params?
    Jeffrey N. Davis
    @penland365
    I’ve gone back and forth. I coded up the Fast Function Call interface to get a feel for how to represent that Request / Response format in different ways, and I haven’t found what “feels” like the simplest solution to that yet
    Finding a nice typesafe solution that’s “obvious” has been more difficult for the Request than the Response
    Fred Jonsson
    @enginoid
    I'm finally back with some more details about the mysterious "Could not find element X in Row" bug: finagle/roc#74
    Jeffrey N. Davis
    @penland365
    Awesome! I'm doing morning stuff with kids, but I will look at that bug report in a couple of hours
    Divan Visagie
    @divanvisagie
    Hi , what is the strategy for prepared statements?
    Jeffrey N. Davis
    @penland365
    Hi @divanvisagie , sorry, was on vacation last week.
    Prepared Statements are something I’ve been working on and around recently, and I hope to dive int it further this month
    Jeffrey N. Davis
    @penland365
    I just pushed a new version, 0.0.4 containg dependency updates - Fiangle 6.36.0, Cats 0.6.0, Circe to 0.5.0-M2
    Vladimir Popov
    @dokwork_twitter

    Hi! How I can set value of argument in the sql query. For example:

    val req = new Request("SELECT * FROM user WHERE name = ?")

    What i shoul do for set "John" as argument of query?