by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 26 2019 21:07
    scala-steward synchronize #139
  • Jan 26 2019 14:33

    mpilquist on v1.11.0

    (compare)

  • Jan 26 2019 14:32

    mpilquist on 1.11.x

    Setting version to 1.11.0 Updated version to 1.11.1-SNAPS… (compare)

  • Jan 25 2019 10:28
    billpcs starred scodec/scodec
  • Jan 22 2019 22:01
    benhanna starred scodec/scodec
  • Jan 21 2019 20:36
    scala-steward opened #139
  • Jan 21 2019 14:51
    mpilquist closed #130
  • Jan 21 2019 14:48
    mpilquist closed #119
  • Jan 21 2019 14:48
    mpilquist commented #119
  • Jan 21 2019 14:48

    mpilquist on 1.11.x

    Scala 2.13.0-M4 Merge branch 'series/1.10.x' in… Upgraded to latest scala-collec… and 2 more (compare)

  • Jan 21 2019 14:48
    mpilquist closed #138
  • Jan 21 2019 14:40
    mpilquist opened #138
  • Jan 21 2019 14:37

    mpilquist on 1.11.x

    Created 1.11 branch (compare)

  • Jan 21 2019 14:31

    mpilquist on xuwei-k

    Upgraded to 2.13.0-M5 (compare)

  • Jan 21 2019 14:19

    mpilquist on xuwei-k

    Scala 2.13.0-M4 Merge branch 'series/1.10.x' in… Upgraded to latest scala-collec… (compare)

  • Jan 21 2019 14:03

    mpilquist on 1.10.x

    Update sbt-scalajs, scalajs-com… Merge pull request #129 from sc… (compare)

  • Jan 21 2019 14:03
    mpilquist closed #129
  • Jan 21 2019 14:03
    mpilquist closed #126
  • Jan 21 2019 14:03

    mpilquist on 1.10.x

    Pad-left on pbcd codec. Fixes #… Using padLeft in pbcd instead o… Merge pull request #133 from lJ… (compare)

  • Jan 21 2019 14:03
    mpilquist closed #133
Soren
@srnb_gitlab
Np! :smile:
Ben Ng
@ben-ng
Hi folks, I was profiling my application and found that 30% of the time is spent just in BitVector.toByteArray. Is there a zero copy or more efficient way to turn many bitvectors into a single byte array? I tried to BitVector.concat the smaller vectors first, but that actually made it slower.
Soren
@srnb_gitlab
You could maybe foldLeft over an empty byte array @ben-ng
Ben Ng
@ben-ng
I don't understand what you mean by that, if the byte array is empty then op would never be run?
Or did you mean use an empty byte array as the initial and then do something different in op
it looks like the time is being spent in compact and go
Soren
@srnb_gitlab
Yeah, use an empty byte array as the initial. I don't know why I said "over" instead of "with an initial" :P
The thing about BitVectors is that they're very lazy
So "flattening" (evaluating) them takes time
And the JVM is not especially good at optimizing bit math
Ben Ng
@ben-ng
Hmm, that just takes me back to the original performance of this. Is there no faster way to write out bit vectors then?
Soren
@srnb_gitlab
I don't know unfortunately, but maybe when a scodec author comes by they can help you
Michael Pilquist
@mpilquist
@ben-ng That seems pretty strange -- can you share a scastie or minimization which shows that behavior? Performance will depend on the type & size of the individual vectors and how they are combined.
Ben Ng
@ben-ng
image.png
Michael Pilquist
@mpilquist
That method is O(nlogn) where n = nodes in tree
Ben Ng
@ben-ng
Ok, I'm new to scodec so this is my first time digging into this. Sounds like reducing the complexity of the encoding would then reduce the number of nodes in the tree and improve the time it takes to serialize it.
Michael Pilquist
@mpilquist
Yeah, though I’m still very curious as to how you’re getting 30% of app performance in that one method. The library is designed to encourage appends of lots of tiny bit vectors and I’ve never seen anythign close to that type of peformance penalty.
Ben Ng
@ben-ng
encoding is taking 17% of app time. toByteArray is taking 70% of app time after I parallelized it. baffling.
Ben Ng
@ben-ng
do each of these become a node in the tree? (int32L :: constant(zeroes(4)) :: int32L :: constant(zeroes(4)))
if so then there are 56 nodes in each tree
Michael Pilquist
@mpilquist
yeah, that’s tiny in the scheme of scodec
Ben Ng
@ben-ng
looks like I'm on scodec 1.x. would 2.x help?
Michael Pilquist
@mpilquist
nah
2.x is for dotty only
Ben Ng
@ben-ng

That method is O(nlogn) where n = nodes in tree

Ah, is this why when I did a BitVector.concat() of the smaller bitvectors before calling a toByteArray, it actually took longer?

yep, math checks out, it took approximately twice as long when I did that
Ben Ng
@ben-ng
So each BitVector has 56 nodes in it. Each node is an int32. It's taking 60us to turn each BitVector into a byte array. Is that normal? That's 15 BitVectors per millisecond.
Gabor Dorcsinecz
@gabor-dorcsinecz
Hi Guys,
Just started with scodec ... maybe it's a trivial question, but I couldn't find anything about it.
I have some bytes coming in, the first 4 bytes has the length, and then I take that many bytes with variableSizeBytes,
but immediately after that, I should apply an other codec sequentially to decode the data part.
This is a very disgusting solution, there must be some other way I can chain codecs! (apply a codec, then apply another)
  val elementCodec: Codec[OSCMessage] = variableSizeBytes(int32, bytes)
    .xmapc { bv => messageCodec.decode(bv.bits).require.value } { msg => messageCodec.encode(msg).require.bytes } //TODO this is horrible
Soren
@srnb_gitlab
@gabor-dorcsinecz variableSizeBytes(int32, messageCodec)
Gabor Dorcsinecz
@gabor-dorcsinecz
Why didn't I think of this :-) Thank you!
Alex Michael Berry
@almibe
Does anyone have any best practices for implementing Ordering for ByteVectors? I'm trying to store ByteVectors in a SortedMap.
Soren
@srnb_gitlab
You can write an Ordering that takes both bytevectors, gets an Iterable from them, and uses the Ordering for Iterables
Michael Pilquist
@mpilquist
There’s a PR open now that adds Ordered for Bit/ByteVector
Alex Michael Berry
@almibe
Okay nice, I'll check out that PR. Thanks both of you.
Alex Michael Berry
@almibe
That seems to work for now. Hopefully it'll get in the next release.
Although I can see the issue with trying to sort ByteVectors. In my case lexicographical ordering is what I want....not sure if that should be the default. For some context I'm trying to mimic what Xodus, RocksDB, FoundationDB, etc. do but with an in-memory model.
Alex Michael Berry
@almibe
Potentially off topic but does anyone have any favorite ways to debug code that does binary encoding/decoding? I can usually use testing frameworks + printlns to figure stuff out but I have a feeling for my current project I'm going to have to make a one off Swing/JavaFX app to help. Just wondering what other people do.
Soren
@srnb_gitlab
I usually am building codecs for formats that other tools generate and whenever I don't get expected behavior I compare my output to what the other tools generate
Lucas Satabin
@satabin
the logToStdOut codec operator is also quite handy when trying to understand exactly where it fails
Alex Michael Berry
@almibe
Thanks logToStdOut sounds useful. In my case I got bit by the old Byte + Byte = Int conversion.
Soren
@srnb_gitlab
@almibe One of the options in sbt-tpolecat will throw a warning on "implicit numeric widening" (which is what you experienced)
I do not remember which
Alex Michael Berry
@almibe
Okay thanks, looking into that plugin and the warnings from tpolecat's blog entries have been on my Scala TODO list, guess I should look into those sooner rather than later. Looks like "-Ywarn-numeric-widen" is what you were mentioning.
Damien O'Reilly
@DamienOReilly

Hi, just for fun and learning, I decided to try solve https://challenge.synacor.com/ (more info https://0bin.net/paste/OSEoQufPgve59sIf#FUukdNbypvDb6XWIFZd+96-3aXeenQ8/thDf4QcsTxp) using Scala, and though scodec would be good to parse the binary file into an instruction-set within my program. I am not sure I am going about decoding the raw byte array in the correct manner. For example some instructions contain pointers, or other data, while other instructions only one 16bit opcode which indicates what to do e.g. HALT, NOOP etc... Therefore I am using case classes and objects as an ADT to model this.

  sealed trait Instruction
  case object Halt extends Instruction
  case class Out(char: Char) extends Instruction
  case class In(char: Char) extends Instruction
  case object Noop extends Instruction

  private val inCodec: Codec[In] = uint16L.as[In]
  private val outCodec: Codec[Out] = uint16L.as[Out]

  private val instructionsCodec: Codec[Instruction] = discriminated[Instruction]
    .by(uint16L)
    .typecase[Out](19, outCodec)
    .typecase[In](20, inCodec)

However I cannot use .typecase with the case objects. Is there any pointers how I should go about this?

Damien O'Reilly
@DamienOReilly
.typecase(21, provide(Noop)) seems to do the trick
Christopher Davenport
@ChristopherDavenport
So... indexOfSlice is quite the gotcha. I think, I assume if I was using scodec proper it does less backtracking. I realized the hard way how often I was iterating over bytevectors byte by byte.
Christopher Davenport
@ChristopherDavenport
@mpilquist Is there any easier way other than me building custom state machines iterating index in this regard?
Alex Michael Berry
@almibe
Is there a good way to override how Codec.encode handles Options when passed a case class? I want it to output nothing if the Option is None and to just output the actual value if it is a Some. I'll never have to decode the resulting value.
1 reply
Artem Nikiforov
@nikiforo

scodec/scodec-stream#74

I've also run scalafmt on projects to pass some CI checks.