Protocol buffer compiler for Scala. Consider sponsoring ScalaPB: https://github.com/sponsors/thesamet
Hey, just wondering, has anyone had success integrating scalapb (or even specifically akka-grpc) with a tool like buf? I saw there were some messages back in March about buf, but I haven't been able to see any of the examples associated with those conversations or ultimately what the outcome was.
That's been reported to work.
It would be interesting to see if ScalaPB can be deployed on BSR. It's not something I looked into yet.
Hi @thesamet did you have a chance to look at thesamet/sparksql-scalapb-test#6 I'm unsure what else I can try from my side and I'd really appreciate your guidance.
Hi @mikedias sorry for missing this. The example seems to run with spark-submit (outside databricks), also the exception you are getting is within databricks, although your code doesn't reference anything about databricks. Could you investigate that? Do you have access to databricks support?
Hello community. My client has a central repo containing all protobufs for messages going out over the Kafka backbone.
Given we are talking 'backbone' infrastructure, the protobufs are used to generate code for several languages, including Go and Python.
We have specified options for ScalaPB, e.g.
syntax="proto3";
package x.y.z.service.some_service;
import "scalapb/scalapb.proto";
import "google/protobuf/timestamp.proto";
import "messages/common/common.proto";
message SomeServiceEvent {
option (scalapb.message).extends = "x.y.z.HasMessageMetadata[SomeServiceEvent]";
option (scalapb.message).companion_extends = "x.y.z.HasKafkaMetadata[SomeServiceEvent]";
...
}but other language (i.e. Python) processors fail with
Import "scalapb/scalapb.proto" was not found or had errors.Is there a way to use language (ScalaPB) specific options that are ignored by other language processors?
Hi @DarrenBishop , this is exactly the scenario auxiliary scoped options are for. You put all these options in a separate proto file and ensure that this file is only available in the ScalaPB build.
for ex: val rawdata : List[Map[String,Any]] = List(Map("deviceid" -> 860267 , "altitude " -> 0.062),Map("deviceid" -> 860268 , "altitude " -> 0.062))
i need to store it into decoded struct message IOTDecodeResult {
google.protobuf.Struct decoded = 1;
}
google.protobuf.Struct decoded = 1;
}
@thesamet fantastic! The auxiliary scoped options worked a treat, thanks.
Can this facility be used to switch a field from being a
Can this facility be used to switch a field from being a
oneof xyz to a oneof sealed_value ?
@thesamet apologies, having read over the docs a number of times now, I realise my last question a) does not make sense and b) won't give me mileage
What I was trying to do was create a sealed oneof (idiomatic-Scala) that can take a case object or a case class i.e. something structurally equivalent to Option[Boolean] with values of None and Some(true).
Unfortunately, sealed oneof does not allow enums:
java.lang.UnsupportedOperationException: This field is not of message type.Is there a trick to generating objects or case objects but extending scalapb.GeneratedMessage?
@thesamet I think the ideal solution would be the ability to specify the name of the Empty case in a sealed oneof to something more meaningful
message Some {
<your-type> = 1;
}
message Option {
oneof sealed_value {
option (scalapb.oneof).empty_name = "None";
Some some = 1;
}
}wdyt?
Hello everybody. I am new to scala and SBT, so please excuse me if this is a newbie question. I am trying to optimize my build time (I have a sbt project) and I noticed that every time i use "compile" new scala classes are generated. Is this expected behavior? Shouldn't the protocGenerate command only be invoked if there are changes in the proto definition file?
I see. Is there something wrong with my configuration perhaps?
ThisBuild / scalaVersion := "2.13.8"
ThisBuild / version := "0.1.0"
ThisBuild / organization := "Jure"
Compile / scalaSource := baseDirectory.value / "chisel4ml" / "scala" / "main"
Compile / unmanagedSourceDirectories += baseDirectory.value / "chisel4ml" / "scala" / "lbir"
Compile / unmanagedSourceDirectories += baseDirectory.value / "chisel4ml" / "scala" / "services"
crossTarget := baseDirectory.value / "bin"
assembly / assemblyJarName := "chisel4ml.jar"
ThisBuild / assemblyMergeStrategy := {
case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
case _ => MergeStrategy.first
}
PB.deleteTargetDirectory := false
PB.additionalDependencies := Nil
Compile / PB.includePaths += file(root.base.getAbsolutePath)
Compile / PB.protoSources := Seq(baseDirectory.value / "chisel4ml" / "lbir")
Compile / PB.targets := Seq(
scalapb.gen(flatPackage = true) -> baseDirectory.value / "chisel4ml" / "scala"
)
val chiselVersion = "3.5.1"
lazy val root = (project in file("."))
.settings(
name := "chisel4ml",
libraryDependencies ++= Seq(
"edu.berkeley.cs" %% "chisel3" % chiselVersion,
//"edu.berkeley.cs" %% "chiseltest" % "0.5.2" % "test",
"edu.berkeley.cs" %% "treadle" % "1.5.3",
"com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf",
"io.grpc" % "grpc-netty" % scalapb.compiler.Version.grpcJavaVersion,
"com.thesamet.scalapb" %% "scalapb-runtime-grpc" % scalapb.compiler.Version.scalapbVersion,
"org.slf4j" % "slf4j-api" % "1.7.5",
"org.slf4j" % "slf4j-simple" % "1.7.5"
),
scalacOptions ++= Seq(
"-language:reflectiveCalls",
"-deprecation",
"-feature",
"-Xcheckinit",
"-P:chiselplugin:genBundleElements",
),
addCompilerPlugin("edu.berkeley.cs" % "chisel3-plugin" % chiselVersion cross CrossVersion.full),
)
@jurevreca12 Unrelated you probably don't need seting
PB.deleteTargetDirectory and PB.additionalDependencies. The latter can cause the generated classes directory get out of sync with the protos in case you remove messages or proto files, removing the latter will allow you to manually remove the dependency on scalapb-runtime since it will get added automatically.
5 replies
Maybe the issue is related to generating into an unmanaged source directory (the scalapb.gen line). Have you tried changing that to managed location as in the install instructions? https://scalapb.github.io/docs/installation/#installing-in-sbt-recommended
Guys, hi! I have a question about supporting scalapb repo in buf BSR. Is it actual right now? Can I use it or should I find another way to link scalapb.proto? I'm talking about https://buf.build/scalapb
Hey @StasSuslov , looks like scalapb repo on buf was set up by someone else, so not officially supported here. I am not familiar with BSR and how it works. It looks like buf is getting some traction recently so I'd like to see how an integration can work. Please share your findings!
I download the demo of scalapb plugin, but the test file can not find the TestMessageFieldNums class in idea. but the sbt shell can work correctlly
If the project gets imported prior to the first run, it can miss the generated code.
1 reply
i creat this demo by "sbt new scalapb/protoc-gen-template.g8"
if i set this fold as source, it work correctlly .Unfortunately， when i re import ,this setting will invalid。
i have set this manually ,it does not work.
However, even without this,
sourceDirectories should have an entry for the src_managed directory.
So it looks like parent
main directory is added automatically which should be sufficient.
So this looks like an intellij bug.
It's probably not the advice you were hoping for, but maybe try vscode with metals on this project. I anticipate this will work flawlessly.
1 reply