Hello! I was just trying out avrohugger (looking to introduce it generally to the organization), but am running into a rather confusing error when I try to generate a schema with enums. Do you have any advice?
val schemaStr =
"""
|{
| "type": "record",
| "name": "Employee",
| "fields": [
| {
| "name": "Gender",
| "type": "enum",
| "symbols": [ "male", "female", "non_binary" ]
| }
| ]
|}
|""".stripMargin
val myScalaTypes = Some(Standard.defaultTypes.copy(enum = EnumAsScalaString))
val generator = Generator(Standard, avroScalaCustomTypes = myScalaTypes)
generator.stringToFile(schemaStr)Which throws:
ToolBoxError: reflective compilation has failed:
identifier expected but string literal found.
...
val myScalaTypes = Some(SpecificRecord.defaultTypes.copy(decimal = ScalaBigDecimal(Some(RoundingMode.HALF_EVEN))))
val generator = Generator(Standard, avroScalaCustomTypes = myScalaTypes), or I'd be curious if you had something else in mind
src_managed. Avrohugger generates into that directory by default, in target, intended for source files managed totally by sbt (and thus are excluded from publishing as well). Overriding that is probably a question for sbt, but I wonder if it'll work for you if you configure Avrohugger's output to use a directory in src, e.g. https://github.com/julianpeeters/sbt-avrohugger/blob/master/src/sbt-test/sbt-avrohugger/overridesettings/build.sbt#L12
avroSpecificScalaSource. https://github.com/julianpeeters/sbt-avrohugger#settings
Hey @julianpeeters avrohugger does not seem to support indirect recursion in schemas despite it being fine in avro. e.g.
{
"type": "record",
"name": "A",
"fields": [
{
"name": "B",
"type": [
"null",
{
"name": "C",
"type": "record",
"fields": [
{
"name": "D",
"type": [
"null",
"A"
]
},
]
}
]
}
]
}Avrohugger is throwing a key not found error.
I did some investigating into the code and I noticed that problem is with the order of type registration and compilation. There is a SourceFormat.registerType method call for each “compilation unit”. I imagine the assumption was that as a topological search is being done first then it would be safe to do registration just before compiling each unit. However in order to support indirect recursion you would have to to register all types first before you can start compiling them.
Would it be possible to change this in order to support indirect recursion?
We need to split Avro models across different sbt projects.
Is it possible with sbt & avro-hugger to use a model from an .avsc in one sbt sub-project, in an .avsc file in another sbt sub-project?
With default settings, avro-hugger keeps complaining that the required type from the other project is not found.
Thx a lot!
2 replies
Hi, I am trying to generate an ADT from this schema using ScalaADT
protocol Foo {
record Bar {
string a;
}
record Baz {
string b;
}
}but what I get is
/** MACHINE-GENERATED FROM AVRO SCHEMA. DO NOT EDIT DIRECTLY */
final case class Bar(a: String)
final case class Baz(b: String)I would have expected this
/** MACHINE-GENERATED FROM AVRO SCHEMA. DO NOT EDIT DIRECTLY */
sealed trait Foo
final case class Bar(a: String) extends Foo
final case class Baz(b: String) extends FooInside the build.sbt file I have inserted
avroScalaSpecificCustomTypes in Compile := {
avrohugger.format.Standard.defaultTypes.copy(
protocol = avrohugger.types.ScalaADT)
}
sourceGenerators in Compile += (avroScalaGenerate in Compile).taskValueDid I omit something?
Is the behavior I have described to you actually what one should expect?
I apologize if it is a known problem and if it is just my misunderstanding, and I hope not to waste your time unnecessarily.
Thank you
And I'm having the first issue:
new MemberData(, , , , , , , , , , )Do you know why it's generating those , ?
It's not compilable code