scodec/scodec

Most discussion is on Typelevel Discord: https://discord.gg/bSQBZA3Ced

mpilquist

@mpilquist:matrix.org

[m]

https://javadoc.io/doc/org.scodec/scodec-core_2.13/latest/index.html

Jakub Kozłowski

@kubukoz:matrix.org

[m]

ah sorry, I meant 2.x :(

and the shiny new scaladoc

mpilquist

@mpilquist:matrix.org

[m]

Oh ha, no, I think scala3doc is still disabled in sbt-spiewak? Or if not, then it should be published

Rad Rares

@radRares1

hello do you have any examples on how should I build a StreamEncoder from scodec-stream? I have a large binary file which I want to encode/decode that contains a huge list of elements, so I would like to be able to stream encode-decode those elements, but I cannot get a hold of how should I approach it. I do have codecs defined for the items in the file which compose a product codec that represent the whole file. Sorry if this is trivial.

mpilquist

@mpilquist:matrix.org

[m]

Generally, use the constructors on the StreamEncoder companion object - e.g., StreamEncoder.many(myCodec).toPipe[IO]

Rad Rares

@radRares1

That's what I first tried, using the example provided in scala-stream for the StreamDecoder. But, it just tells me that the symbol many cannot be resolved and as I went inside to see what StreamEncoder contained, indeed the companion object had only the apply and instance methods. Maybe I imported the wrong thing? though I added to my build file the latest dependency from the releases page.

mpilquist

@mpilquist:matrix.org

[m]

hm what version are you using?

Rad Rares

@radRares1

1.2.0 and the StreamEncoder I found is in a package named scodec.stream.encode and there's a a trait StreamEncoder and an associated object there.

mpilquist

@mpilquist:matrix.org

[m]

oh that's why, 1.2 vs 2.x

the scodec.org releases page is out of date with versions -- really need to get that site cleaned up

Rad Rares

@radRares1

that was the issue!! I really thought I was looking in the wrong jar but it wouldn't make sense, didn't think to check the sonatype for the versions. thank you so much for the help!

Sander Dijkhuis

@sander

Hi! I’m trying to write a codec for macaroons:

macaroon: VERSION opt_location IDENTIFIER EOS caveats EOS SIGNATURE

caveats: caveat caveats
       |

caveat: opt_location IDENTIFIER opt_vid EOS

I’ve got:

  private val caveat: Codec[Caveat] =
    "caveat" | (optionalLocation :: identifier :: optionalVerificationKeyId <~ endOfSection)
      .as[Caveat]
  private val caveats: Codec[Vector[Caveat]] =
    "caveats" | vector(caveat)
  val macaroonV2: Codec[Macaroon] =
    (version ~> optionalLocation :: identifier :: endOfSection ~> caveats :: endOfSection ~> authenticationTag)
      .as[Macaroon]

but when trying to decode, the vector codec fails upon endOfSection right before the authenticationTag. How do I find/create a variant of vector that just returns the result so far and remaining bits as soon as its codec.decode fails?

Sander Dijkhuis

@sander

Could imagine something like

  private val caveats: Codec[Vector[Caveat]] = "caveats" | lazily {
    choice(caveat :: caveats, provide(Vector.empty))
  }

but caveat :: caveats of course does not xmap into a Codec[Vector[Caveat]]

Sander Dijkhuis

@sander

I think I’ve got it:

  def seeWhatHappensVector[A](codec: Codec[A]): Codec[Vector[A]] =
    Codec[Vector[A]](
      Encoder.encodeSeq(codec.asEncoder)(_),
      (bits: BitVector) => {
        @tailrec def helper(rest: BitVector,
                            acc: Vector[A]): Attempt[DecodeResult[Vector[A]]] =
          codec.decode(rest) match {
            case Successful(DecodeResult(v, rem)) => helper(rem, v +: acc)
            case Failure(_)                       => successful(DecodeResult(acc, rest))
          }
        helper(bits, Vector.empty)
      }
    )

Sander Dijkhuis

@sander

(s/v +: acc/acc :+ v)

Brandon Stilson

@bbstilson

Hey folks. Dumb question, but how do I install the library? There are no instructions on the github or the official documentation site.

Ah, I found it under releases.

Brandon Stilson

@bbstilson

I am trying to decode a fixed amount of bytes, into N number of elements that are variably sized. Concretely, I am trying to write a parser for ID3v2 MP3 tags. There is a header which tells you how many bytes to decode, then there are N "frames", each with their own header which tells you how many bytes to decode. I've got the lower level parser, but I'm struggling to write the higher level one (list of N). I don't think listOfN will work because I don't know how many frames there are ahead of time.

For the frame content, I wrote a codec like so:

def fromHeader(header: TagFrameHeader): Codec[TagFrameContent] =
  fixedSizeBytes(header.size.size, utf8)
    .xmap(TagFrameContent.apply, (x: TagFrameContent) => x.content)

which is used for the Frame codec like so:

TagFrameHeader.codec.flatZip(TagFrameContent.fromHeader)

What's the best way to construct N of these based on bytes?

Brandon Stilson

@bbstilson

Sander Dijkhuis has almost exactly what I need...uncanny.

mpilquist

@mpilquist:matrix.org

[m]

Semi-regularly note that the website and docs are out of date and I'd love some help with them :)

I'm hoping to revamp the site when Scala 3's built in site & doc generator is ready and well integrated with SBT

sken

@sken77

is there a more idiomatic way to do this

        codecs.utf8_32.xmap[UUID]( str => 
            scodec.codecs.uuid
              .decodeValue(BitVector.fromValidBase58(str))
              .require,
          _.toString
        )

sken

@sken77

codecs.utf8_32.exmapc(str => {
      for {
        bits <- Attempt.fromEither(BitVector.fromBase58Descriptive(str).left.map(Err(_)))
        uuid <- scodec.codecs.uuid.decodeValue(bits)
      } yield uuid
    })( uuid => Attempt.Successful(uuid.toString))

came up with this

Rad Rares

@radRares1

I'm running into an Insufficient number of elements error from a listOfN decoder when using a StreamDecoder on a file. However, if I bump up the chunkSize passed to fs2.io.file.readAll enough (~8MB), then the error seems to go away. Is this behavior expected?

hello, I'm curious if this was patched, I'm running into the same thing. currently using 1.11.7 for scodec-core and 2.0.1 for scodec-stream

ziggystar

@ziggystar

Hi. Is there something like an empty codec that consumes no input and returns Unit? I have a problem, where an HList is expected, but I have only a single codec. So I could write myCodec :: Codec.zero. Or how would I go about lifting a Codec into an HList? Because I also cannot do myCodec :: HListCodec.hnilCodec, since hnilCodec is inaccessible.

mpilquist

@mpilquist:matrix.org

[m]

myCodec.hlist converts a Codec[A] to Codec[A :: HNil]

is that what you need?

ziggystar

@ziggystar

@mpilquist:matrix.org Thank you. That's what I was looking for.

cranst0n

@cranst0n

@mpilquist:matrix.org Any plans to update/publish scodec-protocols to the latest fs2, etc?

mpilquist

@mpilquist:matrix.org

[m]

Oh huh, I thought that was published already

I guess not

Oliver Schrenk

@oschrenk

What is the the idiomatic way of building a custom Codec? For example: I want to have a Codec[Info] that consumes 13 bytes and turns it into an Info class.

Currently I'm mapping over the tuples, which has can be difficult with a bigger number of byte-structures (but that might be something that is unavoidable, but then I am also getting a GenCodec back which I'm not sure about.

val ChainInfoCodec: GenCodec[(((Boolean, Int), Int), Int), Info] = (bool(8) ~ int32 ~ int32 ~ int32).map {
  case (((enabled, min), center), max) =>
    Info(enabled, min, center, max)
}

The API does offer narrow, and widen . narrow does give me a proper Codec back. So at another place in my code I do,

val CountCodec: Codec[Int] =
  (int32 ~ ignore(8)).narrow(a => Attempt.successful(a._1 - 1), c => (c, ()))

But then there is also xmap and exmap which look they could be helpful.

Adam Luchjenbroers

@AdamLuchjenbroers

I'm a bit baffled by an error message I'm getting:

Type mismatch. Required: PictureHeader => Codec[L_], found: PictureHeader => Codec[Vector[Long]]

My code is as follows:

  case class PictureHeader(width: Int, height: Int, offsetX: Int, offsetY: Int)
  case class Picture(header : PictureHeader, columnOffsets : Vector[Long])

  def headerCodec : Codec[PictureHeader] = {
    ("width" | uint16L) :: ("height" | uint16L) ::
      ("offsetX" | int16L) :: ("offsetY" | int16L)
  }.as[PictureHeader]

  def pictureCodec : Codec[Picture] = {
    ("header" | headerCodec).flatPrepend { header =>
      "columnOffsets" | vectorOfN(provide(header.width), uint32L)
    }
  }.as[Picture]

I'm baffled, because I can't figure out why it's expecting a Codec[L_], or even what an L_ is supposed to be.

Adam Luchjenbroers

@AdamLuchjenbroers

Ok, weird - it works as expected if I use flatZipHList instead of flatPrepend. I'm still new to Scodec so I'm not sure what the difference is here.

Adam Luchjenbroers

@AdamLuchjenbroers

@oschrenk Is this example code what you're looking for? https://github.com/scodec/scodec/blob/main/unitTests/src/test/scala/scodec/examples/MpegPacketExample.scala

Adam Luchjenbroers

@AdamLuchjenbroers

On a slightly different note - are there any features / calls that can work from file offsets / seek to a particular location? Or is the focus of Scodec largely on handling streams?

Michael Pilquist

@mpilquist

Hi everyone! Periodic reminder that discussion has moved to the Typelevel Discord: https://discord.gg/bSQBZA3Ced

Oliver Schrenk

@oschrenk

@oschrenk Is this example code what you're looking for? https://github.com/scodec/scodec/blob/main/unitTests/src/test/scala/scodec/examples/MpegPacketExample.scala

No. I can create scodec codec. I am more looking for the part how to generate Javascript code from scodec and then on how to use it. Maybe less on the code site, but more on project setup side.

Daniel Sebban

@dsebban_twitter

Hi, I have a general question. I have been using Apache Arrow for a few weeks now and really like the 0 copy deserialization feature, meaning it doesn’t allocate the object and reuses magically the bytes . Is it the same when deriving a codec from a case class using scodec, meaning deserializing as almost 0 overhead ?

Joseph Denman

@JosephDenman

Hi everyone.

I'm using scodec to decode bits coming from a socket into messages. My read looks like this:

Stream
        .repeatEval(socket.read(receiveBufferSize, readTimeoutOpt))
        .unNoneTerminate >>= Stream.chunk)
        .through(StreamDecoder.many(Decoder[In]).toPipeByte)

My decoder looks like this:

Attempt.fromTry(Try(VMQuery.parseFrom(bits.toByteArray.drop(4))).map(DecodeResult(_, BitVector.empty)))

I'm using protobuf, so I'm dropping the first four bytes of the message. For small messages, this works fine, but when I get a bigger message, the Attempt in the above expression evaluates to:

Failure(While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.)

My understanding is that the many method will attempt to decode messages repeatedly, and if one can't be decoded, read more bits until it succeeds. The bits that represent the message it's failing on don't all come in at once, so it looks like the decoding is failing. But it's not retrying again once the bits arrive. Any suggestions?

staffanu

@staffanu

Hi I'm migrating to scodec 2.x (since it supports scala 3) and cannot figure out how to handle coproducts. Codec.coproduct is gone since the fork, and I can't figure out what to do instead. All the docs I've found seems to describe the 1.x way?

Michael Pilquist

@mpilquist

@staffanu There's no coproduct support in 2.x, partially because there's no coproduct defined in shapeless-3

I thought about creating our own scodec coproduct, modelled after shapeless 2, but then thought it might be better to just use things like DiscriminatorCodec, which gives up some type safety in exchange for a simpler programming model

BTW, most discussion is on Discord these days

staffanu

@staffanu

Thanks -- and yes, I realized that Discord is where to ask after asking. I managed to get it working by manually listing all subtypes of my trait using discriminated ... by ... typecase.

Where communities thrive

People

Repo info

Activity