Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Julian Peeters
    @julianpeeters
    @zhoekstra_gitlab whoops, I mean like
    avroScalaCustomTypes in Compile := {
      avrohugger.format.Standard.defaultTypes.copy(
        enum = ScalaCaseObjectEnum)
    }
    Zach Hoekstra
    @zhoekstra_gitlab
    That works! Thanks for updating that. I know how difficult making effective README's are. Glad I could help!
    Kostya Golikov
    @lazyval
    hey @julianpeeters! If I want to stick to vanilla case classes, but still need an access to schema sometimes (when I persist the data), is there any good way to get it? I want to stay away from SpecificRecord for several reasons
    Julian Peeters
    @julianpeeters
    Hi @lazyval , how about in the companion object, would that work for you? Try
    avroScalaCustomTypes in Compile := {
      avrohugger.format.Standard.defaultTypes.copy(
        record = ScalaCaseClassWithSchema)
    }
    Kostya Golikov
    @lazyval

    Seems like it works the way I want! Thanks a lot, Julian!

    In the mean time, I've sketched a bug fix for one of the bugs, would love to hear your opinion on it

    julianpeeters/avrohugger#108

    Kostya Golikov
    @lazyval

    the doc mentions

    Standard Vanilla case classes (for use with Apache Avro's GenericRecord API, etc.)

    but I am a bit puzzled how I can shovel those to generic record API. Can you come up with an example?

    1. I generated case classes from schema
    2. I have some Avro-encoded data files
    3. ???
    4. I mapped Avro-encoded files to case class representation

    in particular, how can I map generated case class to GenericRecord?

    Kostya Golikov
    @lazyval
    This message was deleted
    Julian Peeters
    @julianpeeters
    Thanks for the PR, @lazyval looking great, added a comment
    lots of examples in the scripted tests, currently living in sbt-avrohugger https://github.com/julianpeeters/sbt-avrohugger/tree/master/src/sbt-test/avrohugger
    Julian Peeters
    @julianpeeters
    Note that Avro4s generates it's own schema that it uses for serialization, which is derived from the Scala that you or avrohugger generates. It would be nice if Avro4s, or another tool, could serialize using the schema avrohugger can put in the companion object
    thecooladvisor
    @thecooladvisor
    Hi, we are using AVRo hugger to generate the Scala case class schema. Since the column count in the Avro file is more than 254 so the Scala file generating is failing with JVM parameter count more than 254 parameters. Is there a way to resolve this. I am heavily using Avrohugger for my project.
    Any help to resolve this issue will be useful
    I have posted in the stack overflow site but since it is critical so any one can help.
    Julian Peeters
    @julianpeeters
    hi @thecooladvisor, there is no workaround via avrohugger since that's a limitation of the JVM. A JVM expert may have a better answer for you, I can only recommend either
    1. not using the JVM for your use case
    2. evolve your data to a schema that will fit in the JVM
    Christopher Davenport
    @ChristopherDavenport
    You could translate such that you start building internal case class/tuple representations at that size.
    Although conditional logic would be hard to adapt to.
    SemanticBeeng
    @SemanticBeeng

    @julianpeeters Do you know an avro client runtime implementation that builds with scalajs ?
    What is the distance between what avrohugger does and such a client runtime?

    Should avrohugger be able to handle something like
    https://github.com/SemanticBeeng/RPC-Avro-Seed/blob/6a2f99ba78356e962d40967d532a79d991db37ab/server/modules/protocol/src/main/resources/PeopleService.avdl#L2-L7

    Just looking for insights from building avrohugger.

    Would you say it would be easy to build such a runtime with hammock ?
    http://pepegar.com/hammock/marshalling.html.

    Thanks

    Ioann German
    @io-german
    @julianpeeters could you please check if sbt-avrohugger is published on maven central? Because looks like it doesn't - https://mvnrepository.com/artifact/com.julianpeeters/sbt-avrohugger returns 404. Or have you moved it to some other repository?
    Julian Peeters
    @julianpeeters
    @io-german I seem to be able to pull the artifact. I wonder if that link ever worked? Maybe this is related? sbt/sbt#3879 Regardless, the badge on the repo links here: https://search.maven.org/artifact/com.julianpeeters/sbt-avrohugger/2.0.0-RC15/jar
    hopefully that'll work for you, not sure what to say otherwise
    Julian Peeters
    @julianpeeters

    @SemanticBeeng no, I don't know of an avro runtime for scalajs. There's the javascript project called AVSC, might be able to make a facade for that. Writing a new runtime library would probably look a lot like circe, and/or like avro4s, which I imagine would then be integrated into hammock like the circe example you linked.

    I'd expect avrohugger will able to generate from- and to- the directories you want

    RPC support is limited to SpecificRecord format, however, otherwise the avro message definitions are ignored. I'd imagine you'd want to use Standard vanilla case classes with Scalajs. I believe 47 degrees uses some post-processing of avrohugger's output in order to handle generating definintions for RPC/message.

    Ioann German
    @io-german
    @julianpeeters yeah, looks like the artifact was indexed but download link on the page you mentioned is not working (direct link from the page https://search.maven.org/remotecontent?filepath=com/julianpeeters/sbt-avrohugger/2.0.0-RC15/sbt-avrohugger-2.0.0-RC15.jar). Search query on mvnrepository.com also doesn't return the plugin (https://mvnrepository.com/search?q=sbt-avrohugger)
    Ioann German
    @io-german
    Ok, I got it - this is neither SBT plugins nor Maven Central. it's Sonatype's repository. Perhaps it's worth to mention in the example that one should add another resolver to avoid such confusion.
    Greg Silin
    @gregsilin

    @julianpeeters et al, I see some previous discussions about supporting joda DateTime. We have hacked avrohugger.matchers.TypeMatcher in our fork to support DateTime in the following fashion:

            case Schema.Type.STRING   =>
              // bit of a HACK: to get joda DateTime to gen.
              // note that "datetime" does not make it into "getLogicalType" since avro (java project 1.8.2) filters out types not in the spec
              if(scala.util.Try(schema.getObjectProp("logicalType")).toOption.getOrElse("") == "datetime")
                RootClass.newClass(nme.createNameType("org.joda.time.DateTime"))
              else if(scala.util.Try(schema.getObjectProp("logicalType")).toOption.getOrElse("") == "localdate")
                RootClass.newClass(nme.createNameType("org.joda.time.LocalDate"))

    (credit goes to my former colleague not in this chat)

    We wanted to start a conversation on what people think on having something like this making into the main repo.

    I know the above looks very hacky, so maybe allowing TypeMatcher to be extendable to support custom types? This way we don't force it on everyone.

    Julian Peeters
    @julianpeeters
    @gregsilin thanks for the good ideas, however I'm going to pass, due to the general deprecation of Joda, and the lack of desire to support another yet another customization system in avrohugger. I don't discourage a fork, if that's a solution you.
    deklanw
    @deklanw
    nvm
    deklanw
    @deklanw
    so I'm wanting to use avrohugger instead of avro-tools to generate classes for use in Flink. I tried to switch to avrohugger, but started getting this error: https://pastebin.com/YnxiUNtX
    maybe a Flink question, but maybe someone here knows
    Julian Peeters
    @julianpeeters

    @deklanw I'm not experienced with Flink, and I don't recognize how that Order could be coming from avrohugger, however the fact that we're seeing a reflection-related error might suggest that indeed the avrohugger-generated Scala classes are the cause. Scala can't be reflected by Java very well, and avro (and frameworks' avro integration) loves to try: https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroInputFormat.java#L116

    I imagine you're trying with the Specific api? I'd recommend sending a PR to Flink to use the avro methods that take a schema as an argument instead of getting it by reflection.

    Alternatively, you could try using the avro Generic api that it looks like Flink also supports. You'd probably want to generate Scala classes as avrohugger's Standard format, and then make a function to convert them into GenericRecords before using them with Flink. You may look into Avro4s to automate that last part, as I believe it uses GenericRecord under the hood, but beware it also regenerates it's own schema (from the class definition, and which usually matches the schema used to generate the class, but not always)

    deklanw
    @deklanw
    @julianpeeters I actually just switched to using Beam via Scio, still using Avrohugger and I got it working with Beam so shrug works for me
    thanks
    deklanw
    @deklanw
    @julianpeeters I'm on sbt-avrohugger 2.0.0-RC15 and this is turning into a Long
        {
          "name": "myField",
          "type": "long",
          "logicalType": "timestamp-millis"
        },
    should it not be an Instant or Timestamp?
    Julian Peeters
    @julianpeeters
    @deklanw no, in avro schemas logicalType is a property of types, not fields. https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types
    Lots of examples in the tests too, in case you'd like to see more https://github.com/julianpeeters/avrohugger/blob/ea3ecbfed4f702d54d185768be0af1d7e61f7c21/avrohugger-core/src/test/avro/logical.avsc#L15-L21
    deklanw
    @deklanw
    @julianpeeters ah, sorry for the silly mistake. thought i was losing it. thanks
    Julian Peeters
    @julianpeeters
    @deklanw nah, not losing it :) seems to be an avro "gotcha". I wish avrohugger could do more to prevent it but were limited by the non-type-checking schema -- a schema idl that typechecked might be a fun and useful project if anyone is interested
    deklanw
    @deklanw
    is there a way to get Joda Instants with sbt-avrohugger?
    Julian Peeters
    @julianpeeters
    no, there's no plan for supporting that. The question was asked a few posts up tho, sounds like they might have a fork
    Josh McDade
    @joshm1

    hi, i'm just getting into avro, so newbie question. Using the sbt plugin to generated case classes. From what I read in the documentation, I thought this idl would generate a case class with a UUID field, but it's still string.

    @namespace ("com.example.directory.event")
    protocol Test {
      record TestRecord {
        @logicalType("uuid") string id;
      }
    }
    // case class TestRecord(id: String)

    What am I missing?

    Julian Peeters
    @julianpeeters
    hey @joshm1 that's due to avro itself not supporting uuid definitions in idls, as of 1.8.2. Kinda a gotcha IMO, I'll try to make it more clear in the README that currently uuid only works in schemas (.avsc)
    Josh McDade
    @joshm1
    Thanks @julianpeeters I guess just assumed IDL and avsc were interchangable definitions
    pascalsAger
    @pascals-ager
    Hello @julianpeeters , Firstly, thank you for the awesome project.
    I just want to know if there is some special configuration needed on Nexus proxy to resolve the plugin dependency.
    addSbtPlugin("com.julianpeeters" % "sbt-avrohugger" % "2.0.0-RC15")
    The artifact gets resolved without my nexus proxy. However, on using the proxy it doesn't resolve.
    Although I can clearly see it here https://repo.maven.apache.org/maven2/com/julianpeeters/sbt-avrohugger_2.12_1.0/2.0.0-RC15/
    the proxy seems to have a problem fetching, and i suspect it is due to the suffix 2.12_1.0. I just wanted to know if anyone else has faced this issue and how they went about solving it..
    pascalsAger
    @pascals-ager
    I was able to resolve the issue by setting the nexus proxy layout to permissive
    Julian Peeters
    @julianpeeters
    thanks a ton @pascals-ager . I'll add a note about that to the readme