Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Tim Chan
@timchan-lumoslabs
If I stick with extending AbstractBijection, how do I use Conversion.asMethod? I'm getting a 'could not find implicit value for parameter conv' error
oh nm. i figured it out!
Sam Ritchie
@sritchie
@timchan-lumoslabs yeah, you just need to make it available in implicit scope
Tim Chan
@timchan-lumoslabs
val foo = Foo("abc", 123)
implicit val inj = SpecificAvroCodecs[Foo]
val bytes = Injection[Foo, Array[Byte]](foo)
val attempt = Injection.invert[Foo, Array[Byte]](bytes)
attempt.get
gives me
attempt: scala.util.Try[com.foo.Foo] = Failure(com.twitter.bijection.InversionFailure: Failed to invert: [B@39675182)
com.twitter.bijection.InversionFailure: Failed to invert: [B@39675182
Caused by: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.foo.Foo
    at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:248)
    at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:189)
I'm using avrohugger to generate SpecificRecord case classes.
anand79
@anand79
hi I am reading from my own kafka topic and getting error - org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -128
Any idea what could be the issue
P. Oscar Boykin
@johnynek
are you using Bijection?
anand79
@anand79
yes
P. Oscar Boykin
@johnynek
what is likely the case is that the data is not in the format you expect.
it was reading, and found a negative length
anand79
@anand79
i am producing the data and consuming it too
using same avro schema
look like my issue is similar to what Tim Chan had - com.twitter.bijection.InversionFailure: Failed to invert: [B@44e0a074
P. Oscar Boykin
@johnynek
failure to invert happens anytime the data is not what you expect.
it may, or may not, be his issue.
his issue looks like avro did not think that his Foo class was a Specific avro class.
step 1, you can in a test generate random objects, serialize and then deserialize them.
get kafka out of the picture
when you are sure the injection is working correctly, you can figure out what is going on with kafka
anand79
@anand79
ok let me try that, thx
that worked
val schema: Schema = new Schema.Parser().parse(scala.io.Source.fromInputStream(getClass.getResourceAsStream(firmLegacySecurityAvroSchemaFile)).mkString)
val recordInjection: Injection[GenericRecord, Array[Byte]] = GenericAvroCodecs.toBinary(schema)

val record = new GenericData.Record(schema)
record.put("id", firmLegacySecurity.id)
record.put("masterSecID", firmLegacySecurity.masterSecID)
record.put("positionID", firmLegacySecurity.positionID)
record.put("dateAdded", firmLegacySecurity.dateAdded.toString)
record.put("dateModified", if(firmLegacySecurity.dateModified != null) firmLegacySecurity.dateModified.toString else "")

val bytes: Array[Byte] = recordInjection.apply(record)

val record1: GenericRecord = recordInjection.invert(bytes).get
val id = record1.get("id").toString
val masterSecId = record1.get("masterSecID").toString
val positionId = record1.get("positionID").toString
P. Oscar Boykin
@johnynek
so, that suggests that something about kafka is the issue
anand79
@anand79
dang, now what could that be :), Thanks for your help
P. Oscar Boykin
@johnynek
Good luck!
anand79
@anand79
Hi, does anyone have example of serialization/deserialization of nested avro schema using bijection?
anand79
@anand79
something like this gives error -
record.put("legacy.MasterSecID", firmLegacySecurity.masterSecID)
ridaamine
@ridaamine
Hey guys
is it possible to convert from a GenericRecord to a case class
ridaamine
@ridaamine
hey guys still me i have an avro GenericRecord and i want to map it to a case class is this possible or do i need to do it manually
johnnywiller
@johnnywiller
Hi guys, how easy/clean is using bijection in java to convert between json <-> byte[]
?
I looked for docs, but there aren't much
Martijn Hoekstra
@martijnhoekstra
should array2Traversable allow or disallow sharing changes to the underlying array? Or is it more of a "if you do that, who knows what might happen" kind of thing?