by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 26 2019 21:07
    scala-steward synchronize #139
  • Jan 26 2019 14:33

    mpilquist on v1.11.0

    (compare)

  • Jan 26 2019 14:32

    mpilquist on 1.11.x

    Setting version to 1.11.0 Updated version to 1.11.1-SNAPS… (compare)

  • Jan 25 2019 10:28
    billpcs starred scodec/scodec
  • Jan 22 2019 22:01
    benhanna starred scodec/scodec
  • Jan 21 2019 20:36
    scala-steward opened #139
  • Jan 21 2019 14:51
    mpilquist closed #130
  • Jan 21 2019 14:48
    mpilquist closed #119
  • Jan 21 2019 14:48
    mpilquist commented #119
  • Jan 21 2019 14:48

    mpilquist on 1.11.x

    Scala 2.13.0-M4 Merge branch 'series/1.10.x' in… Upgraded to latest scala-collec… and 2 more (compare)

  • Jan 21 2019 14:48
    mpilquist closed #138
  • Jan 21 2019 14:40
    mpilquist opened #138
  • Jan 21 2019 14:37

    mpilquist on 1.11.x

    Created 1.11 branch (compare)

  • Jan 21 2019 14:31

    mpilquist on xuwei-k

    Upgraded to 2.13.0-M5 (compare)

  • Jan 21 2019 14:19

    mpilquist on xuwei-k

    Scala 2.13.0-M4 Merge branch 'series/1.10.x' in… Upgraded to latest scala-collec… (compare)

  • Jan 21 2019 14:03

    mpilquist on 1.10.x

    Update sbt-scalajs, scalajs-com… Merge pull request #129 from sc… (compare)

  • Jan 21 2019 14:03
    mpilquist closed #129
  • Jan 21 2019 14:03
    mpilquist closed #126
  • Jan 21 2019 14:03

    mpilquist on 1.10.x

    Pad-left on pbcd codec. Fixes #… Using padLeft in pbcd instead o… Merge pull request #133 from lJ… (compare)

  • Jan 21 2019 14:03
    mpilquist closed #133
Soren
@srnb_gitlab
Nope, that's how you should do it @Voltir, although I think you've got your methods backwards
Nick Childers
@Voltir
?
Soren
@srnb_gitlab

So

00 00 00 04

is empty data?

Nick Childers
@Voltir
yea
i guess
i should test that
Soren
@srnb_gitlab

Or

00 00 00 01 00 00 00 01 field

is 1 byte of field data?

Nick Childers
@Voltir
the fields are variable length but non zero
Soren
@srnb_gitlab
If it's the former, you need to switch your xmap, if it's the latter, you can keep it as it is
I.e. I'm not understanding whether you meant the byte count includes the bytes used in the byte count or excludes the bytes used in the field count
Nick Childers
@Voltir
i actually guess empty would be 00 00 06 00 00
because its composed with a vectorOfN(int16)
Soren
@srnb_gitlab
Ok, so yeah, your xmap should be _ + 4, _ - 4
Nick Childers
@Voltir
so <total bytes> <number of fields> <field> <field> ...
Soren
@srnb_gitlab
and total bytes counts itself?
Nick Childers
@Voltir
yep
Soren
@srnb_gitlab
So
Bah, I got my stuff backwards, you're right
Nick Childers
@Voltir
ok, good - because my test worked, lol :)
Soren
@srnb_gitlab
When decoding, you subtract 4, and when encoding you add 4: _ - 4, _ + 4
Nick Childers
@Voltir
right
cool - well, i like scodec so far :)
ty for your help
Soren
@srnb_gitlab
Be on the lookout for scodec 2 coming to scala 3, it's gonna be so much cooler
Np! :smile:
Ben Ng
@ben-ng
Hi folks, I was profiling my application and found that 30% of the time is spent just in BitVector.toByteArray. Is there a zero copy or more efficient way to turn many bitvectors into a single byte array? I tried to BitVector.concat the smaller vectors first, but that actually made it slower.
Soren
@srnb_gitlab
You could maybe foldLeft over an empty byte array @ben-ng
Ben Ng
@ben-ng
I don't understand what you mean by that, if the byte array is empty then op would never be run?
Or did you mean use an empty byte array as the initial and then do something different in op
it looks like the time is being spent in compact and go
Soren
@srnb_gitlab
Yeah, use an empty byte array as the initial. I don't know why I said "over" instead of "with an initial" :P
The thing about BitVectors is that they're very lazy
So "flattening" (evaluating) them takes time
And the JVM is not especially good at optimizing bit math
Ben Ng
@ben-ng
Hmm, that just takes me back to the original performance of this. Is there no faster way to write out bit vectors then?
Soren
@srnb_gitlab
I don't know unfortunately, but maybe when a scodec author comes by they can help you
Michael Pilquist
@mpilquist
@ben-ng That seems pretty strange -- can you share a scastie or minimization which shows that behavior? Performance will depend on the type & size of the individual vectors and how they are combined.
Ben Ng
@ben-ng
image.png
Michael Pilquist
@mpilquist
That method is O(nlogn) where n = nodes in tree
Ben Ng
@ben-ng
Ok, I'm new to scodec so this is my first time digging into this. Sounds like reducing the complexity of the encoding would then reduce the number of nodes in the tree and improve the time it takes to serialize it.
Michael Pilquist
@mpilquist
Yeah, though I’m still very curious as to how you’re getting 30% of app performance in that one method. The library is designed to encourage appends of lots of tiny bit vectors and I’ve never seen anythign close to that type of peformance penalty.
Ben Ng
@ben-ng
encoding is taking 17% of app time. toByteArray is taking 70% of app time after I parallelized it. baffling.
Ben Ng
@ben-ng
do each of these become a node in the tree? (int32L :: constant(zeroes(4)) :: int32L :: constant(zeroes(4)))
if so then there are 56 nodes in each tree
Michael Pilquist
@mpilquist
yeah, that’s tiny in the scheme of scodec
Ben Ng
@ben-ng
looks like I'm on scodec 1.x. would 2.x help?
Michael Pilquist
@mpilquist
nah
2.x is for dotty only
Ben Ng
@ben-ng

That method is O(nlogn) where n = nodes in tree

Ah, is this why when I did a BitVector.concat() of the smaller bitvectors before calling a toByteArray, it actually took longer?

yep, math checks out, it took approximately twice as long when I did that