ScalaPB/community

Protocol buffer compiler for Scala. Consider sponsoring ScalaPB: https://github.com/sponsors/thesamet

Nadav Samet

@thesamet

@adriansnep16 you mean having primitive types (ints, strings) and not just messages? Scala wouldn't allow it since all the types that extend a sealed trait must be defined in the same file, and ints and strings are already defined elsewhere.

@frekw I have limited internet connectivity in the coming week, if needed feel free to file an issue on GitHub and I'll get back to it.

adriansnep16

@adriansnep16

no i mean a structure like this

message foo {
string cause = 1;
}

message bar {
string data = 1;
}

message my_trait {
string id = 1;
oneof sealed_value {
foo fooEl = 2;
bar barEl = 3;
}
}

this will fail at compile time because of "string id" which is present in "mytrait" because "oneof sealed.." doesn't allow additional fields ....and my question is if there is a possibility of modeling "my_trait" to also contain the "string id" field

Nadav Samet

@thesamet

I think that what you want is to create another message MyTraitWithId which would have two fields: string id and 'my_trait mt'

Fredrik Wärnsberg

@frekw

@thesamet will do :pray:

Karan Shah

@karanssh

Hi folks! I am trying to debug a weird issue I am running into. I have a scala grpc server, and clients in go and java. I can connect from the go, java clients fine on ubuntu where I build all of them, but when I compile them on a centOS box and try, the scala grpc server spins up, accepts connections but never responds. Any pointers on how I could debug this would be appreciated

Nadav Samet

@thesamet

Hi @Karen, scalapb uses grpc-java under the hood. It sounds like the issue is unrelated to scalapb, given that it happens when you switch operating systems of the non-scala clients. I suggest to reproduce it with an example have server and client and ask on a grpc help forum.

Karan Shah

@karanssh

Thanks @thesamet , I'll take a look!

Grant

@gavares

@thesamet - I’m working on an MR to remove the deprecation warnings from the zio2 branch. The first test TestServiceSpec.serverStreamingSuite is failing and I’m having a hard time figuring out why. The test is expecting a response stream with two eleemnts Response(“X1”), Response(“X2”) but the Stream only contains the first element. I’ve done a bit of debugging and the stream is exiting successfully. I can’t seem to figure out why the 2nd stream element is never included in the response.

I’ll keep working on it but the draft MR is here if you have any insights: scalapb/zio-grpc#311

Grant

@gavares

I did manage to fix one of the failing tests that was previously failing on the zio2 branch

Grant

@gavares

Managed to fix the failing tests. MR updated.

Raymond Te Hok

@rtehok

hi there, I am using scalapb-runtime-grpc_2.13-0.11.5.jar and it currently raises CVE errors such as CVE-2017-7860, CVE-2017-7861... does any one know why this is raised and how to fix it ?

5 replies

Linh Nguyen

@tuleism

@thesamet in case you missed it scalapb/zio-grpc#303. Let me know if it's not the way to do it!

Nadav Samet

@thesamet

@tuleism merged

Linh Nguyen

@tuleism

Thanks!

Peter Zhong

@jiujiu1123

Hi Has anyone seem errors like this?

[error] Unspecified value parameter message.
[error]             __inner = Option(__inner.fold(_root_.scalapb.LiteParser.readMessage[livongo.microservice.util.filtering.Testing.FilterTarget.Nested](_input__))(_root_.scalapb.LiteParser.readMessage(_input__, _)))
[error]                                                                                                                                                 ^
[error] /Users/peter.zhong/Desktop/platform-libraries/microservice-util/target/scala-2.13/src_managed/test/scalapb/livongo/microservice/util/filtering/Testing/FilterTarget.scala:433:127: not enough arguments for method readMessage: (input: com.google.protobuf.CodedInputStream, message: livongo.microservice.util.filtering.Testing.FilterTarget.Nested)(implicit cmp: scalapb.GeneratedMessageCompanion[livongo.microservice.util.filtering.Testing.FilterTarget.Nested]): livongo.microservice.util.filtering.Testing.FilterTarget.Nested.
[error] Unspecified value parameter message.
[error]             __values += _root_.scalapb.LiteParser.readMessage[livongo.microservice.util.filtering.Testing.FilterTarget.Nested](_input__)
[error]

Nadav Samet

@thesamet

@jiujiu1123 I haven't seen, but can you check you are generating code using the same version of ScalaPB that you use to compile? (compilerplugin and scalapb-runtime versions need to match)

Peter Zhong

@jiujiu1123

Thanks let me verify that

Peter Zhong

@jiujiu1123

@thesamet could it be because the protoc-gen is a different minor version?

Nadav Samet

@thesamet

no, that's fine.

Looks like both compielrplugin and scalapb-runtime are on 0.10.x

Peter Zhong

@jiujiu1123

Yeah which is confusing since the versions should be matching.

Nadav Samet

@thesamet

I am not sure what the list in the screenshot represents, the compilerplugin is not a runtime dependency.

try to figure out why you patch versions are different

another possibility is that from some reason, you still have source code that generated by a different version of the compilerplugin that is configured right now. Try to perform a sbt clean, and have it regenerate.

Peter Zhong

@jiujiu1123

Let me try that. Really appreciate all the advice.

Peter Zhong

@jiujiu1123

I got it to work. Thank you so much!

Nadav Samet

@thesamet

:thumbsup:

Alexis BRENON

@AlexisBRENON

Hi. I try to use scalapb-spark, to read a dataset from Json. It seems to have a problem when a "repeated" field does not appear in the json stream...
In the "messageReads" method, the __fieldsMap.get(...) call returns Some(null)... Is it a known behavior ?

Alexis BRENON

@AlexisBRENON

For example, from this proto file:

syntax = "proto3";

package com.adyoulike.data.proto;

message MyRepeated {
  repeated string ListField = 1;
}

I can load data from this jsonl file:

{"ListField": ["foo", "bar"]}
{"ListField": []}

But this one

{"ListField": ["foo", "bar"]}
{"ListField": []}
{}

fails with

Exception in thread "main" java.lang.NullPointerException
    at com.adyoulike.data.proto.repeated.MyRepeated$.$anonfun$messageReads$4(MyRepeated.scala:84)
    at scala.Option.map(Option.scala:230)
    at com.adyoulike.data.proto.repeated.MyRepeated$.$anonfun$messageReads$1(MyRepeated.scala:84)

(that is to say the __fieldMap.get(...) line

I use an old version of spark and scalapb, but I cannot upgrade for the moment...
I'll try to workaround, using a custom type for my Seq: Option[Seq[...]]

1 reply

Alexis BRENON

@AlexisBRENON

Other question related to scalapb-spark. Is it possible to mix Encoders. I mean, I have a proto message Message. I define a case class case class Nesting(msg: Message, foo: Int). How can I provide an encoder for this case class ?

Nadav Samet

@thesamet

@AlexisBRENON can you give a code shippet on how you read the json and turn it into a case class? I couldn't tell if scalapb-json4s is involved.

1 reply

@AlexisBRENON Recent versions of sparksql-scalapb use frameless to derive encoders. See the tests for OuterCaseClass and OuterCaseClassTimestamp to see an example. For the second case class it uses frameless injections to customize the timestamp type: https://github.com/scalapb/sparksql-scalapb/blob/4017c0c2dea0e261e731547fbdee1202a2e61b16/sparksql-scalapb/src/test/scala/PersonSpec.scala#L34-L35

1 reply

Ángel Cervera Claudio

@angelcervera

Hi @thesamet I need to release a new version of osm4scala. At the moment, I'm using com.thesamet:sbt-protoc:0.99.34 and com.thesamet.scalapb:compilerplugin:0.10.2
Do you think that make sense to upgrade to latests versions? I'm looking for a changelog, but looks like it is not available, right?
Are new versions adding better performance and bug fixes or only new features (That I'm not using)?

Nadav Samet

@thesamet

Hi @angelcervera , the CHANGELOG is in https://github.com/scalapb/ScalaPB/blob/master/CHANGELOG.md . v0.11.x contains additional features, bug fixes, performance improvements, and Scala 3 support.

Ángel Cervera Claudio

@angelcervera

Great

Nadav Samet

@thesamet

@AlexisBRENON , I had a chance looking at it today. It looks like when columns are missing in the json input, spark puts in nulls for the missing value. ScalaPB expects empty arrays and not nulls when parsing a dataframe. I think it will be useful to have ScalaPB accept nulls when parsing spark dataframes. You could modify your input sources or prepare it for consumption by ScalaPB by replacing the nulls. If supporting nulls is something that would be useful for your project, feel free to file an issue with sparksql-scalapb.

5 replies

Nadav Samet

@thesamet

@AlexisBRENON A different approach to try would be to read the input source as String, then use ProtoSQL.udf to use scalapb-json4s to parse the json string into a message case class.

Josip Grgurica

@jkobejs

Hi, scalapb-json4s parser accepts both json name and original field name when deserializing json to proto message
scalapb-circe doesn't have support for that, you can by using preservingProtoFieldNames flag pick if you want to parse using original field or json name, but you cannot do both

I think that it makes sense to have the same behavior in scalapb-circe as in scalapb-json4s, it is easier to provide support for some legacy Json apis where you need to support original names together with new ones

let me know if you're ok with having it in scalapb-circe repo and I'll add support for it :)

Nadav Samet

@thesamet

Hi @jkobejs , scalapb-circe is maintained by @xuwei-k - I anticipate that a PR for it to catch up with scalapb-json4s behavior will be accepted. I suggest to file an issue on that repo.

Josip Grgurica

@jkobejs

Ok, tnx 🙂

Where communities thrive

People

Repo info

Activity