Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    thecooladvisor
    @thecooladvisor
    Hi, we are using AVRo hugger to generate the Scala case class schema. Since the column count in the Avro file is more than 254 so the Scala file generating is failing with JVM parameter count more than 254 parameters. Is there a way to resolve this. I am heavily using Avrohugger for my project.
    Any help to resolve this issue will be useful
    I have posted in the stack overflow site but since it is critical so any one can help.
    Julian Peeters
    @julianpeeters
    hi @thecooladvisor, there is no workaround via avrohugger since that's a limitation of the JVM. A JVM expert may have a better answer for you, I can only recommend either
    1. not using the JVM for your use case
    2. evolve your data to a schema that will fit in the JVM
    Christopher Davenport
    @ChristopherDavenport
    You could translate such that you start building internal case class/tuple representations at that size.
    Although conditional logic would be hard to adapt to.
    SemanticBeeng
    @SemanticBeeng

    @julianpeeters Do you know an avro client runtime implementation that builds with scalajs ?
    What is the distance between what avrohugger does and such a client runtime?

    Should avrohugger be able to handle something like
    https://github.com/SemanticBeeng/RPC-Avro-Seed/blob/6a2f99ba78356e962d40967d532a79d991db37ab/server/modules/protocol/src/main/resources/PeopleService.avdl#L2-L7

    Just looking for insights from building avrohugger.

    Would you say it would be easy to build such a runtime with hammock ?
    http://pepegar.com/hammock/marshalling.html.

    Thanks

    Ioann German
    @io-german
    @julianpeeters could you please check if sbt-avrohugger is published on maven central? Because looks like it doesn't - https://mvnrepository.com/artifact/com.julianpeeters/sbt-avrohugger returns 404. Or have you moved it to some other repository?
    Julian Peeters
    @julianpeeters
    @io-german I seem to be able to pull the artifact. I wonder if that link ever worked? Maybe this is related? sbt/sbt#3879 Regardless, the badge on the repo links here: https://search.maven.org/artifact/com.julianpeeters/sbt-avrohugger/2.0.0-RC15/jar
    hopefully that'll work for you, not sure what to say otherwise
    Julian Peeters
    @julianpeeters

    @SemanticBeeng no, I don't know of an avro runtime for scalajs. There's the javascript project called AVSC, might be able to make a facade for that. Writing a new runtime library would probably look a lot like circe, and/or like avro4s, which I imagine would then be integrated into hammock like the circe example you linked.

    I'd expect avrohugger will able to generate from- and to- the directories you want

    RPC support is limited to SpecificRecord format, however, otherwise the avro message definitions are ignored. I'd imagine you'd want to use Standard vanilla case classes with Scalajs. I believe 47 degrees uses some post-processing of avrohugger's output in order to handle generating definintions for RPC/message.

    Ioann German
    @io-german
    @julianpeeters yeah, looks like the artifact was indexed but download link on the page you mentioned is not working (direct link from the page https://search.maven.org/remotecontent?filepath=com/julianpeeters/sbt-avrohugger/2.0.0-RC15/sbt-avrohugger-2.0.0-RC15.jar). Search query on mvnrepository.com also doesn't return the plugin (https://mvnrepository.com/search?q=sbt-avrohugger)
    Ioann German
    @io-german
    Ok, I got it - this is neither SBT plugins nor Maven Central. it's Sonatype's repository. Perhaps it's worth to mention in the example that one should add another resolver to avoid such confusion.
    Greg Silin
    @gregsilin

    @julianpeeters et al, I see some previous discussions about supporting joda DateTime. We have hacked avrohugger.matchers.TypeMatcher in our fork to support DateTime in the following fashion:

            case Schema.Type.STRING   =>
              // bit of a HACK: to get joda DateTime to gen.
              // note that "datetime" does not make it into "getLogicalType" since avro (java project 1.8.2) filters out types not in the spec
              if(scala.util.Try(schema.getObjectProp("logicalType")).toOption.getOrElse("") == "datetime")
                RootClass.newClass(nme.createNameType("org.joda.time.DateTime"))
              else if(scala.util.Try(schema.getObjectProp("logicalType")).toOption.getOrElse("") == "localdate")
                RootClass.newClass(nme.createNameType("org.joda.time.LocalDate"))

    (credit goes to my former colleague not in this chat)

    We wanted to start a conversation on what people think on having something like this making into the main repo.

    I know the above looks very hacky, so maybe allowing TypeMatcher to be extendable to support custom types? This way we don't force it on everyone.

    Julian Peeters
    @julianpeeters
    @gregsilin thanks for the good ideas, however I'm going to pass, due to the general deprecation of Joda, and the lack of desire to support another yet another customization system in avrohugger. I don't discourage a fork, if that's a solution you.
    deklanw
    @deklanw
    nvm
    deklanw
    @deklanw
    so I'm wanting to use avrohugger instead of avro-tools to generate classes for use in Flink. I tried to switch to avrohugger, but started getting this error: https://pastebin.com/YnxiUNtX
    maybe a Flink question, but maybe someone here knows
    Julian Peeters
    @julianpeeters

    @deklanw I'm not experienced with Flink, and I don't recognize how that Order could be coming from avrohugger, however the fact that we're seeing a reflection-related error might suggest that indeed the avrohugger-generated Scala classes are the cause. Scala can't be reflected by Java very well, and avro (and frameworks' avro integration) loves to try: https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroInputFormat.java#L116

    I imagine you're trying with the Specific api? I'd recommend sending a PR to Flink to use the avro methods that take a schema as an argument instead of getting it by reflection.

    Alternatively, you could try using the avro Generic api that it looks like Flink also supports. You'd probably want to generate Scala classes as avrohugger's Standard format, and then make a function to convert them into GenericRecords before using them with Flink. You may look into Avro4s to automate that last part, as I believe it uses GenericRecord under the hood, but beware it also regenerates it's own schema (from the class definition, and which usually matches the schema used to generate the class, but not always)

    deklanw
    @deklanw
    @julianpeeters I actually just switched to using Beam via Scio, still using Avrohugger and I got it working with Beam so shrug works for me
    thanks
    deklanw
    @deklanw
    @julianpeeters I'm on sbt-avrohugger 2.0.0-RC15 and this is turning into a Long
        {
          "name": "myField",
          "type": "long",
          "logicalType": "timestamp-millis"
        },
    should it not be an Instant or Timestamp?
    Julian Peeters
    @julianpeeters
    @deklanw no, in avro schemas logicalType is a property of types, not fields. https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types
    Lots of examples in the tests too, in case you'd like to see more https://github.com/julianpeeters/avrohugger/blob/ea3ecbfed4f702d54d185768be0af1d7e61f7c21/avrohugger-core/src/test/avro/logical.avsc#L15-L21
    deklanw
    @deklanw
    @julianpeeters ah, sorry for the silly mistake. thought i was losing it. thanks
    Julian Peeters
    @julianpeeters
    @deklanw nah, not losing it :) seems to be an avro "gotcha". I wish avrohugger could do more to prevent it but were limited by the non-type-checking schema -- a schema idl that typechecked might be a fun and useful project if anyone is interested
    deklanw
    @deklanw
    is there a way to get Joda Instants with sbt-avrohugger?
    Julian Peeters
    @julianpeeters
    no, there's no plan for supporting that. The question was asked a few posts up tho, sounds like they might have a fork
    Josh McDade
    @joshm1

    hi, i'm just getting into avro, so newbie question. Using the sbt plugin to generated case classes. From what I read in the documentation, I thought this idl would generate a case class with a UUID field, but it's still string.

    @namespace ("com.example.directory.event")
    protocol Test {
      record TestRecord {
        @logicalType("uuid") string id;
      }
    }
    // case class TestRecord(id: String)

    What am I missing?

    Julian Peeters
    @julianpeeters
    hey @joshm1 that's due to avro itself not supporting uuid definitions in idls, as of 1.8.2. Kinda a gotcha IMO, I'll try to make it more clear in the README that currently uuid only works in schemas (.avsc)
    Josh McDade
    @joshm1
    Thanks @julianpeeters I guess just assumed IDL and avsc were interchangable definitions
    pascalsAger
    @pascals-ager
    Hello @julianpeeters , Firstly, thank you for the awesome project.
    I just want to know if there is some special configuration needed on Nexus proxy to resolve the plugin dependency.
    addSbtPlugin("com.julianpeeters" % "sbt-avrohugger" % "2.0.0-RC15")
    The artifact gets resolved without my nexus proxy. However, on using the proxy it doesn't resolve.
    Although I can clearly see it here https://repo.maven.apache.org/maven2/com/julianpeeters/sbt-avrohugger_2.12_1.0/2.0.0-RC15/
    the proxy seems to have a problem fetching, and i suspect it is due to the suffix 2.12_1.0. I just wanted to know if anyone else has faced this issue and how they went about solving it..
    pascalsAger
    @pascals-ager
    I was able to resolve the issue by setting the nexus proxy layout to permissive
    Julian Peeters
    @julianpeeters
    thanks a ton @pascals-ager . I'll add a note about that to the readme
    Nick Aldwin
    @NJAldwin
    :wave: hello! We’re starting to use sbt-avrohugger. The plugin itself works great to generate scala. However, I’m running into a confusing error.
    If I start with a clean, tests-passing tree of our master branch, without the plugin, then I add sbt-avrohugger to plugins.sbt with no other changes, suddenly the tests in my submodule are unable to resolve any dependencies
    that is, literally the diff is just the addSbtPlugin and nothing else, and suddenly tests are not happy
    Any ideas what could be going on here? I’m using 2.0.0-RC16 if that helps
    happy to open a ticket too but figured I could try here first
    repro:
    1. sbt the-submodule/clean the-submodule/test succeeds
    2. Add addSbtPlugin line to plugins.sbt (there are multiple other plugins already there, working fine)
    3. sbt the-submodule/clean the-submodule/test fails with no dependencies seemingly able to resolve
      (i.e. even scalatest isn’t there, and we get value should is not a member of String for a ”foo” should “bar” in {)
    Nick Aldwin
    @NJAldwin
    I did a little more digging, and it seems to work fine in RC15 — so this is due to some change between RC15 and RC16
    Julian Peeters
    @julianpeeters
    @NJAldwin wow, bizarre, thank you very much for the report. This makes me realize that avrohugger indeed doesn't have a test with submodules. I'll see if I can reproduce this.
    from RC15 to RC16, I tried to match java avro by adding a way to check the classpath during compilation -- something seems to have very wrong!
    I expect I can look more closely tomorrow
    Nick Aldwin
    @NJAldwin
    Thanks, I appreciate your looking into it!