scodec/scodec

Scala combinator library for working with binary data

mpilquist

@mpilquist:matrix.org

[m]

https://javadoc.io/doc/org.scodec/scodec-core_2.13/latest/index.html

Jakub Kozłowski

@kubukoz:matrix.org

[m]

ah sorry, I meant 2.x :(

and the shiny new scaladoc

mpilquist

@mpilquist:matrix.org

[m]

Oh ha, no, I think scala3doc is still disabled in sbt-spiewak? Or if not, then it should be published

Rad Rares

@radRares1

hello do you have any examples on how should I build a StreamEncoder from scodec-stream? I have a large binary file which I want to encode/decode that contains a huge list of elements, so I would like to be able to stream encode-decode those elements, but I cannot get a hold of how should I approach it. I do have codecs defined for the items in the file which compose a product codec that represent the whole file. Sorry if this is trivial.

mpilquist

@mpilquist:matrix.org

[m]

Generally, use the constructors on the StreamEncoder companion object - e.g., StreamEncoder.many(myCodec).toPipe[IO]

Rad Rares

@radRares1

That's what I first tried, using the example provided in scala-stream for the StreamDecoder. But, it just tells me that the symbol many cannot be resolved and as I went inside to see what StreamEncoder contained, indeed the companion object had only the apply and instance methods. Maybe I imported the wrong thing? though I added to my build file the latest dependency from the releases page.

mpilquist

@mpilquist:matrix.org

[m]

hm what version are you using?

Rad Rares

@radRares1

1.2.0 and the StreamEncoder I found is in a package named scodec.stream.encode and there's a a trait StreamEncoder and an associated object there.

mpilquist

@mpilquist:matrix.org

[m]

oh that's why, 1.2 vs 2.x

the scodec.org releases page is out of date with versions -- really need to get that site cleaned up

Rad Rares

@radRares1

that was the issue!! I really thought I was looking in the wrong jar but it wouldn't make sense, didn't think to check the sonatype for the versions. thank you so much for the help!

Sander Dijkhuis

@sander

Hi! I’m trying to write a codec for macaroons:

macaroon: VERSION opt_location IDENTIFIER EOS caveats EOS SIGNATURE

caveats: caveat caveats
       |

caveat: opt_location IDENTIFIER opt_vid EOS

I’ve got:

  private val caveat: Codec[Caveat] =
    "caveat" | (optionalLocation :: identifier :: optionalVerificationKeyId <~ endOfSection)
      .as[Caveat]
  private val caveats: Codec[Vector[Caveat]] =
    "caveats" | vector(caveat)
  val macaroonV2: Codec[Macaroon] =
    (version ~> optionalLocation :: identifier :: endOfSection ~> caveats :: endOfSection ~> authenticationTag)
      .as[Macaroon]

but when trying to decode, the vector codec fails upon endOfSection right before the authenticationTag. How do I find/create a variant of vector that just returns the result so far and remaining bits as soon as its codec.decode fails?

Sander Dijkhuis

@sander

Could imagine something like

  private val caveats: Codec[Vector[Caveat]] = "caveats" | lazily {
    choice(caveat :: caveats, provide(Vector.empty))
  }

but caveat :: caveats of course does not xmap into a Codec[Vector[Caveat]]

Sander Dijkhuis

@sander

I think I’ve got it:

  def seeWhatHappensVector[A](codec: Codec[A]): Codec[Vector[A]] =
    Codec[Vector[A]](
      Encoder.encodeSeq(codec.asEncoder)(_),
      (bits: BitVector) => {
        @tailrec def helper(rest: BitVector,
                            acc: Vector[A]): Attempt[DecodeResult[Vector[A]]] =
          codec.decode(rest) match {
            case Successful(DecodeResult(v, rem)) => helper(rem, v +: acc)
            case Failure(_)                       => successful(DecodeResult(acc, rest))
          }
        helper(bits, Vector.empty)
      }
    )

Sander Dijkhuis

@sander

(s/v +: acc/acc :+ v)

Brandon Stilson

@bbstilson

Hey folks. Dumb question, but how do I install the library? There are no instructions on the github or the official documentation site.

Ah, I found it under releases.

Brandon Stilson

@bbstilson

I am trying to decode a fixed amount of bytes, into N number of elements that are variably sized. Concretely, I am trying to write a parser for ID3v2 MP3 tags. There is a header which tells you how many bytes to decode, then there are N "frames", each with their own header which tells you how many bytes to decode. I've got the lower level parser, but I'm struggling to write the higher level one (list of N). I don't think listOfN will work because I don't know how many frames there are ahead of time.

For the frame content, I wrote a codec like so:

def fromHeader(header: TagFrameHeader): Codec[TagFrameContent] =
  fixedSizeBytes(header.size.size, utf8)
    .xmap(TagFrameContent.apply, (x: TagFrameContent) => x.content)

which is used for the Frame codec like so:

TagFrameHeader.codec.flatZip(TagFrameContent.fromHeader)

What's the best way to construct N of these based on bytes?

Brandon Stilson

@bbstilson

Sander Dijkhuis has almost exactly what I need...uncanny.

mpilquist

@mpilquist:matrix.org

[m]

Semi-regularly note that the website and docs are out of date and I'd love some help with them :)

I'm hoping to revamp the site when Scala 3's built in site & doc generator is ready and well integrated with SBT

sken

@sken77

is there a more idiomatic way to do this

        codecs.utf8_32.xmap[UUID]( str => 
            scodec.codecs.uuid
              .decodeValue(BitVector.fromValidBase58(str))
              .require,
          _.toString
        )

sken

@sken77

codecs.utf8_32.exmapc(str => {
      for {
        bits <- Attempt.fromEither(BitVector.fromBase58Descriptive(str).left.map(Err(_)))
        uuid <- scodec.codecs.uuid.decodeValue(bits)
      } yield uuid
    })( uuid => Attempt.Successful(uuid.toString))

came up with this

Rad Rares

@radRares1

I'm running into an Insufficient number of elements error from a listOfN decoder when using a StreamDecoder on a file. However, if I bump up the chunkSize passed to fs2.io.file.readAll enough (~8MB), then the error seems to go away. Is this behavior expected?

hello, I'm curious if this was patched, I'm running into the same thing. currently using 1.11.7 for scodec-core and 2.0.1 for scodec-stream

ziggystar

@ziggystar

Hi. Is there something like an empty codec that consumes no input and returns Unit? I have a problem, where an HList is expected, but I have only a single codec. So I could write myCodec :: Codec.zero. Or how would I go about lifting a Codec into an HList? Because I also cannot do myCodec :: HListCodec.hnilCodec, since hnilCodec is inaccessible.

mpilquist

@mpilquist:matrix.org

[m]

myCodec.hlist converts a Codec[A] to Codec[A :: HNil]

is that what you need?

ziggystar

@ziggystar

@mpilquist:matrix.org Thank you. That's what I was looking for.

cranst0n

@cranst0n

@mpilquist:matrix.org Any plans to update/publish scodec-protocols to the latest fs2, etc?

mpilquist

@mpilquist:matrix.org

[m]

Oh huh, I thought that was published already

I guess not

Oliver Schrenk

@oschrenk

What is the the idiomatic way of building a custom Codec? For example: I want to have a Codec[Info] that consumes 13 bytes and turns it into an Info class.

Currently I'm mapping over the tuples, which has can be difficult with a bigger number of byte-structures (but that might be something that is unavoidable, but then I am also getting a GenCodec back which I'm not sure about.

val ChainInfoCodec: GenCodec[(((Boolean, Int), Int), Int), Info] = (bool(8) ~ int32 ~ int32 ~ int32).map {
  case (((enabled, min), center), max) =>
    Info(enabled, min, center, max)
}

The API does offer narrow, and widen . narrow does give me a proper Codec back. So at another place in my code I do,

val CountCodec: Codec[Int] =
  (int32 ~ ignore(8)).narrow(a => Attempt.successful(a._1 - 1), c => (c, ()))

But then there is also xmap and exmap which look they could be helpful.

Where communities thrive

People

Repo info

Activity