by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
P. Oscar Boykin
@johnynek
like, you could just do: (ClassTag<SpecificRecordBase>)
Benjamin
@bdparrish
the extends is already defined in the class
P. Oscar Boykin
@johnynek
as long as you keep the clazz being passed in.
Benjamin
@bdparrish
using ClassTag<SpecificRecordBase> got me the other error...
Exception in thread "main" java.lang.InstantiationException
    at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at com.twitter.bijection.avro.SpecificAvroCodecs$.toBinary(AvroCodecs.scala:106)
    at com.twitter.bijection.avro.SpecificAvroCodecs.toBinary(AvroCodecs.scala)
Michael Hsueh
@mkhsueh
Howdy- nice work on Bijection! Currently using it for Avro handling; has anyone gotten this error deserializing Joda DateTime? java.lang.Long cannot be cast to org.joda.time.DateTime
was able to serialize a record with this value: {"name": "eventTime", "type": {"type": "long", "logicalType": "timestamp-millis"}}, but calling injection.invert(the serialized byte[]).get() throws that error
Michael Hsueh
@mkhsueh
this is my injection object private static Injection<MyModel, byte[]> injection = SpecificAvroCodecs.toBinary(scala.reflect.ClassTag$.MODULE$.apply(MyModel.class)); Thanks for any advice!
Michael Hsueh
@mkhsueh
dug into the codebase...( DateBijectionLaws.scala ) looks like round trips were expected using 'String'. Changing the underlying Avro schema to type "string" worked. Seems like it should've had dual support given the logical type?
P. Oscar Boykin
@johnynek
@mkhsueh I have not seen this error, but please file an issue and maybe we can find a fix.
Marius Soutier
@mariussoutier
@mkhsueh Might this be related to the fact that bijection uses Avro 1.7 and you are using features from Avro 1.8?
Tim Chan
@timchan-lumoslabs
I'd like to make a bijection between two case classes that I have. Do I just extend AbstractBijection?
P. Oscar Boykin
@johnynek
@timchan-lumoslabs yes. That's one way. There are also macros that can do this for you for case classes. On the phone now, but should not be too hard to find.
Tim Chan
@timchan-lumoslabs
If I stick with extending AbstractBijection, how do I use Conversion.asMethod? I'm getting a 'could not find implicit value for parameter conv' error
oh nm. i figured it out!
Sam Ritchie
@sritchie
@timchan-lumoslabs yeah, you just need to make it available in implicit scope
Tim Chan
@timchan-lumoslabs
val foo = Foo("abc", 123)
implicit val inj = SpecificAvroCodecs[Foo]
val bytes = Injection[Foo, Array[Byte]](foo)
val attempt = Injection.invert[Foo, Array[Byte]](bytes)
attempt.get
gives me
attempt: scala.util.Try[com.foo.Foo] = Failure(com.twitter.bijection.InversionFailure: Failed to invert: [B@39675182)
com.twitter.bijection.InversionFailure: Failed to invert: [B@39675182
Caused by: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.foo.Foo
    at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:248)
    at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:189)
I'm using avrohugger to generate SpecificRecord case classes.
anand79
@anand79
hi I am reading from my own kafka topic and getting error - org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -128
Any idea what could be the issue
P. Oscar Boykin
@johnynek
are you using Bijection?
anand79
@anand79
yes
P. Oscar Boykin
@johnynek
what is likely the case is that the data is not in the format you expect.
it was reading, and found a negative length
anand79
@anand79
i am producing the data and consuming it too
using same avro schema
look like my issue is similar to what Tim Chan had - com.twitter.bijection.InversionFailure: Failed to invert: [B@44e0a074
P. Oscar Boykin
@johnynek
failure to invert happens anytime the data is not what you expect.
it may, or may not, be his issue.
his issue looks like avro did not think that his Foo class was a Specific avro class.
step 1, you can in a test generate random objects, serialize and then deserialize them.
get kafka out of the picture
when you are sure the injection is working correctly, you can figure out what is going on with kafka
anand79
@anand79
ok let me try that, thx
that worked
val schema: Schema = new Schema.Parser().parse(scala.io.Source.fromInputStream(getClass.getResourceAsStream(firmLegacySecurityAvroSchemaFile)).mkString)
val recordInjection: Injection[GenericRecord, Array[Byte]] = GenericAvroCodecs.toBinary(schema)

val record = new GenericData.Record(schema)
record.put("id", firmLegacySecurity.id)
record.put("masterSecID", firmLegacySecurity.masterSecID)
record.put("positionID", firmLegacySecurity.positionID)
record.put("dateAdded", firmLegacySecurity.dateAdded.toString)
record.put("dateModified", if(firmLegacySecurity.dateModified != null) firmLegacySecurity.dateModified.toString else "")

val bytes: Array[Byte] = recordInjection.apply(record)

val record1: GenericRecord = recordInjection.invert(bytes).get
val id = record1.get("id").toString
val masterSecId = record1.get("masterSecID").toString
val positionId = record1.get("positionID").toString
P. Oscar Boykin
@johnynek
so, that suggests that something about kafka is the issue
anand79
@anand79
dang, now what could that be :), Thanks for your help
P. Oscar Boykin
@johnynek
Good luck!
anand79
@anand79
Hi, does anyone have example of serialization/deserialization of nested avro schema using bijection?
anand79
@anand79
something like this gives error -
record.put("legacy.MasterSecID", firmLegacySecurity.masterSecID)
ridaamine
@ridaamine
Hey guys
is it possible to convert from a GenericRecord to a case class
ridaamine
@ridaamine
hey guys still me i have an avro GenericRecord and i want to map it to a case class is this possible or do i need to do it manually
johnnywiller
@johnnywiller
Hi guys, how easy/clean is using bijection in java to convert between json <-> byte[]
?
I looked for docs, but there aren't much
Martijn Hoekstra
@martijnhoekstra
should array2Traversable allow or disallow sharing changes to the underlying array? Or is it more of a "if you do that, who knows what might happen" kind of thing?