Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 06 18:21
    scala-steward opened #1275
  • Dec 06 18:20
    scala-steward opened #153
  • Dec 04 05:03

    mergify[bot] on master

    Update auxlib, javalib, nativel… (compare)

  • Dec 04 05:03
    mergify[bot] closed #143
  • Dec 04 04:59
    scala-steward synchronize #143
  • Dec 04 01:28

    mergify[bot] on master

    Update sbt-scalafmt to 2.4.5 (#… (compare)

  • Dec 04 01:28
    mergify[bot] closed #1274
  • Dec 04 01:21

    mergify[bot] on master

    Update sbt-scalafmt to 2.4.5 (#… (compare)

  • Dec 04 01:21
    mergify[bot] closed #230
  • Dec 04 01:13

    mergify[bot] on master

    Update sbt-scalafmt to 2.4.5 (#… (compare)

  • Dec 04 01:13
    mergify[bot] closed #139
  • Dec 04 01:11

    mergify[bot] on master

    Update sbt-scalafmt to 2.4.5 (#… (compare)

  • Dec 04 01:11
    mergify[bot] closed #233
  • Dec 04 01:11

    mergify[bot] on master

    Update sbt-scalafmt to 2.4.5 (#… (compare)

  • Dec 04 01:11
    mergify[bot] closed #152
  • Dec 04 01:09
    mergify[bot] labeled #230
  • Dec 04 01:09
    mergify[bot] assigned #230
  • Dec 04 01:09
    scala-steward opened #230
  • Dec 04 01:09
    mergify[bot] labeled #135
  • Dec 04 01:09
    mergify[bot] assigned #135
Quynh Anh "Emma" Nguyen
@qaemma

Hi, I'm implementing tracing with zio-grpc and zio-telemetry. On server side, I'm able to use ZTransform to read the propagated context and start a new Span, since it wrap the whole effect.

On the client side, do we have something similar? Currently I'm using ZClientInterceptor, which is convenient because I have access to the MethodDescriptor and SafeMetadata. With that, I'm able to propagate context by updating SafeMetadata. However, I cannot start a Span inside ZClientInterceptor because it doesn't wrap the call from client to server. If I start a Span like: GreeterClient.sayHello(...).span("sayHello", ...), then I need to manually put it in each call and I don't have access to the MethodDescriptor.

Nadav Samet
@thesamet
Hi @qaemma , I've just looked into it and ended up implementing transform for client traits. It's a bit tricky for client-streaming and bidi calls (so that's left unimplemented), but I got it for unary (regular) and server-streaming. Will you be able to try it out by depending on the latest snapshot version 0.5.1+7-ada03de2-SNAPSHOT ? The change is scalapb/zio-grpc@6eb9949 . Let me know if this works - would be great if you can help with sending a PR with unit tests for this as well :)
6 replies
Mike Dias
@mikedias
Hello @thesamet, thank you for answering this question, the solution you provided is pretty close, I just had trouble finding the correct Spark Encoder for GenericMessage...
Nadav Samet
@thesamet
@mikedias Sure! Responded in SO.
Mike Dias
@mikedias
Thank you @thesamet, really appreciate it! I have a follow up question if you don't mind. I really need to step up my knowledge about Scala types... :)
Mike Dias
@mikedias

So, I took a look in the TypedEncodersMessageTypedEncoder as inspiration and implemented a GeneratedMessageTypedEncoder inspired by it, replacing ct.runtimeClass with cmp.defaultInstance.getClass and it works fine on my small example:

trait TypedEncoders extends FromCatalystHelpers with ToCatalystHelpers with Serializable {

  class GeneratedMessageTypedEncoder(implicit
    cmp: GeneratedMessageCompanion[_ <: GeneratedMessage]
  ) extends TypedEncoder[GeneratedMessage] {
    override def nullable: Boolean = false

    override def jvmRepr: DataType = ObjectType(cmp.defaultInstance.getClass)

    override def catalystRepr: DataType = protoSql.schemaFor(cmp.scalaDescriptor)

    def fromCatalyst(path: Expression): Expression = {
      val expr = pmessageFromCatalyst(cmp, path)

      val reads = Invoke(
        Literal.fromObject(cmp),
        "messageReads",
        ObjectType(classOf[Reads[_]]),
        Nil
      )

      val read = Invoke(reads, "read", ObjectType(classOf[Function[_, _]]))

      Invoke(read, "apply", ObjectType(cmp.defaultInstance.getClass), expr :: Nil)
    }

    override def toCatalyst(path: Expression): Expression = {
      messageToCatalyst(cmp, path)
    }
  }
}

object GeneratedMessageTypedEncoder {

  val typedEncoders: TypedEncoders = new TypedEncoders {
    @transient
    lazy val protoSql = ProtoSQL
  }

  def encoder()(implicit
    cmp: GeneratedMessageCompanion[_ <: GeneratedMessage]
  ): typedEncoders.GeneratedMessageTypedEncoder = {
    new typedEncoders.GeneratedMessageTypedEncoder()
  }
}

and then:

  import scalapb.spark.Implicits._

  implicit val entityCompanion = EntityMapper.typeUrlToCompanion(typeUrlFromRuntime)
  implicit val encoder = GeneratedMessageTypedEncoder.encoder()

  spark.read.format("text")
    .load("/tmp/protobuf/notification").as[String]
    .map { s => Base64.getDecoder.decode(s) }
    .map { bytes => Envelope.parseFrom(bytes) }
    .map { envelope => envelope.getEntity.unpack(entityCompanion) }
    .show()

Do you see any problems with this approach?

Nadav Samet
@thesamet
@mikedias just responded in SO. reading here!
It looks like the code snippet pins one specific companion (the one that corresponds typeUrlFromRuntime) and always uses that.
If the entire input consists of the same type, then it should be fine.
I think that the GeneratedMessageTypedEncoder is not necessary, in the SO I suggested to use implicitly to obtain the encoder on your own (and store that in the map)
Mike Dias
@mikedias
yes, will be filtering records of the same type before transforming them. I'll play with implicitly in the map. Thanks heaps @thesamet!
Ondra Pelech
@sideeffffect

Hello everybody!
How can I solve the problem of generating code for other languages besides Scala (Python in particular) from a proto spec that contains scalapb.options? I have

...
import "scalapb/scalapb.proto";

option (scalapb.options) = {
  flat_package: true
  enum_value_naming: CAMEL_CASE
  enum_strip_prefix: true
  bytes_type: "scodec.bits.ByteVector"
};
...

in my proto file, because I like this better than ScalaPB's defaults. But what do I do when a generate a Python code from this? The Python generator doesn't have scalapb/scalapb.proto, so it doesn't know scalapb.options and so it fails. Is there some way around this problem?

Nadav Samet
@thesamet
@sideeffffect you can either add a copy of scalapb.proto and generate python code for it on your own. Alternatively, you can exclude the package-scoped options proto file from your Python build. One easy way to do it, is to just move it to a different directory that is private to your scala build, and add this directory as list of proto sources in SBT: Compile / PB.protoSources += file("path/to/private-dir")
Alexis Hernandez
@AlexITC

Hi everybody, I got an interesting use case and I'm wondering if there is a better way to handle it.

I got a scalajs project that depends on protobuf sources which are coming from an npm package, I ended up writing this custom logic so that scalapb doesn't run until the sources are pulled, hooking the resolution into the artifactResolver task seems like a hack.

Compile / PB.artifactResolver := (Compile / PB.artifactResolver)
  .dependsOn(Compile / npmInstallDependencies)
  .value

Compile / PB.protoSources := {
  Seq(
    (npmUpdate / crossTarget).value / "scalajs-bundler" / "main" / "node_modules" /  "custom-protos"
  )
}

Please let me know if there is a better way, thanks

Nadav Samet
@thesamet
Hi @AlexITC , looks reasonable. I think using protocGenerate instead of artifactResolver would make more sense. protocGenerate is the main protoc task where the proto file gathering happens. The protocGenerate task depends on artifactResolver, so both would be technically correct.
Alexis Hernandez
@AlexITC
thanks for the hint, I think I need to upgrade scalapb because such task doesn't exist in my current project:
value protocGenerate is not a member of object sbtprotoc.ProtocPlugin.autoImport.PB
Nadav Samet
@thesamet
Alexis Hernandez
@AlexITC
Actually, that was my first try but for some reason it didn't worked
Nadav Samet
@thesamet
Got it. As long as it works this way, should be fine.
Ondra Pelech
@sideeffffect

Thanks @thesamet for the tip. I tried it, but it doesn't seem to work, so I'm probably doing something wrong. I have split it like this

// src/main/protobuf/Contract.proto

message MyMessage {
  string my_field = 1;
}
message OtherMessage {
  bytes other_field = 1;
  MyMessage my_message = 2;
}

and

// src/main/protobuf-scala/ScalaSpecific.proto

import "scalapb/scalapb.proto";

option (scalapb.options) = {
  flat_package: true
  enum_value_naming: CAMEL_CASE
  enum_strip_prefix: true
  bytes_type: "scodec.bits.ByteVector"
};

And in by sbt config

Compile / PB.protoSources += file((Compile / sourceDirectory).value + "/protobuf-scala")

But the provided scalapb.options options don't seem to have any effect on src/main/protobuf/Contract.proto. The seem to apply only on the messages that I put in src/main/protobuf-scala/ScalaSpecific.proto. Is there a way how to achieve that? Or some other workaround?

Brice Jaglin
@bjaglin
@sideeffffect i think you are missing the scope field to indicate that the scalapb.options apply to a specific package (and not just the file where they are declared, the default), see https://scalapb.github.io/docs/customizations/#package-scoped-options
Ondra Pelech
@sideeffffect

That was it @bjaglin ! This now works as I need

// src/main/protobuf-scala/ScalaSpecific.proto
import "scalapb/scalapb.proto";
option (scalapb.options) = {
  scope: PACKAGE
  flat_package: true
  enum_value_naming: CAMEL_CASE
  enum_strip_prefix: true
  bytes_type: "scodec.bits.ByteVector"
};

Thank you :heart: :heart: :heart:

Remi Guittaut
@remiguittaut
Hi guys, I have a question, has any of you ever had the use case to host the grpc server under a base path? I want my server to expose grpc under xxx.com/rpc
The only thing we have on ServerBuilder is forPort
Remi Guittaut
@remiguittaut
Ha, and another question, is it possible to plug the http backend? I'm using zio-grpc, also based on netty, so it would be nice to use 1 single http layer
Nadav Samet
@thesamet
@remiguittaut for the first question, it would be better to ask grpc-java folks. ScalaPB/zio-grpc uses it directly for building the server and for handling request parsing. For the second question, I suspect the answer is no, due to limitation in grpc-java. Maybe your app can host it on two different ports and use nginx, envoy or another proxy to make it appear on the same port from the outside. Using a proxy might also solve your first question.
Alexandre Lebrun
@xela85
Hello !
I have recently discovered prototool and buf and would like to enable one of these linters in my schemas repository. I have initiated a naive CI by running buf on proto resources. Unfortunately import "scalapb/scalapb.proto"; is unresolved and leads to compilation errors. How do you circumvent these types of errors (same problem in the IJ plugin) ? Thanks in advance :)
Philipp Hoffmann
@philipphoffmann
Hey there! We could need some help using ScalaPB together with Spark. Basically our question is, how do we deal with protobuf extensions in Spark? We figured out already that ScalaPB generates Scala case classes for our protobuf extensions just fine. But deserializing our binaries in Spark gives us a DataFrame with the schema inferred for our root message only (without any extension fields). I think this is more or less expected since we use something like RootMessage.parseFrom(...) to deserialize which only gives us the RootMessage. So i guess my question is, how can we deserialize the extension fields? Thx for you help :)
Nadav Samet
@thesamet
@xela85 add a copy of scalapb/scalapb.proto somewhere to your source tree or environment and provide the path to the parent directory (one containing scalapb as a subdirectory) to the search path used by those linters.
@philipphoffmann Currently there's no support for extensions in sparksql-scalapb. Theoretically there could be many extensions to any message so the schema can be pretty large if we take the union. There's also no facility to discover all extensions for a given message. However, if there's a specific extension you are expecting you could do { payload => val msg = RootMessage.parseFrom(payload); (msg, msg.extension(ext))} to return both the message and the extension you expect as a tuple (or define a new case class), and transform it as a dataframe from there.
Philipp Hoffmann
@philipphoffmann
@thesamet excellent Nadav. Thats exactly our missing piece to the puzzle :) thx a lot!
Alexandre Lebrun
@xela85
Perfect, thank you :)
Mike Dias
@mikedias
Hello, it seems that the sparksql-scalapb artifact doesn't have the latest 5 versions published. Is that expected?
Nadav Samet
@thesamet
Hi @mikedias , there are no unpublished artifacts. I assume you expect the version to match the ScalaPB release version, but those two libraries are released independently. The versions of sparksql-scalapb and scalapb need to match according to the table here https://scalapb.github.io/docs/sparksql/#setting-up-your-project
Rohit Ramprasad
@rrampr

Hello this might be a simple question but I don't see a straightforward way around it . ScalaPBC generates case classes from proto definitions. The regular protobuf compiler can generate Java classes (that inherit from com.google.protobuf.GeneratedMessageV3 and implements com.google.protobuf.Message).

If Im writing a function that can naively operate on both generated proto classes, how would I do so? There is no common ancestor or interface I can use.
Reason I want this: Writing a small internal library in java that will be used within scala services as well as java services. The input to one of my functions is gonna be a protobuf type. I can't use Message as the type of the argument as it wouldn't work from a scala service

Nadav Samet
@thesamet
@rrampr You can write two functions. They can even have the same name since the parameter types are going to be different.
1 reply
There is nothing that abstracts over the two protobuf types. If you want to minimize duplication you can use the typeclass pattern to have an implicit typeclass provided that would provide the abstractions you need over the two proto types.
Timothy Bess
@tdbgamer

Hey, so is there a generic way to reference the Java class associated with a generated Scala proto?

I wanted to provide an interface that takes A and B that are GeneratedMessages and automatically serialize to/from Java protos and provides a Scala only interface to the user. Currently I have to do:
abstract class MyInterface[A, AJava, B, BJava](implicit aJavaSupport: JavaProtoSupport[A, AJava], bJavaSupport: JavaProtoSupport[B, BJava])
Which works, but it's annoying cause the user has to fill out a giant type signature. Ideally they shouldn't have to care about or reference the Java classes. Any ideas?

Nadav Samet
@thesamet
Hi @tdbgamer , I think this can be improved if we introduce the java type as a dependent type in the JavaProtoSupport. Can you provide additional context on what you are trying to enable for the users? What methods will you have in this abstract class? If the intent is to make it easier to convert Java to Scala protos why it needs two types (A and B)?
Timothy Bess
@tdbgamer
I'm making an internal wrapper around Kafka where each service takes an input proto and returns an output proto. I basically want those services to only have to care about Scala protos, and I will transparently convert to/from java protos when I write to or read from Kafka.
Timothy Bess
@tdbgamer

https://pastebin.com/n0UbrFum

^ This is the base class

essentially uses the JavaProtoSupport to create RecordSerializers and RecordDeserializers for the scala protos
Nadav Samet
@thesamet
Got it. Makes sense. Does Kafka care about the Java protos at all? I haven't worked with it much, but I thought it can accept arbitrary payloads (arrays of bytes)
Nadav Samet
@thesamet
@tdbgamer if we want MyInterface to only depend on A and B, we could introduce some typeclass like JavaProtoSupport[A<:GeneratedMessage]. Sounds like what you need in it is: parseFromJava(a: A#JavaType): A and scalaToJava(a: A): A#JavaType. Any other requirements?
6 replies
Timothy Bess
@tdbgamer

It can, but schema registry uses the Java protos (I assume through the file descriptor) to track changes to schema, ensure backwards compatibility, and allow for dynamically deserializing protobuf messages.

But there are lots of tools in the Kafka ecosystem that need to dynamically deserialize messages on topics, and to do that with protobuf, you need to have the schema of the message that was published since protobuf doesn't encode type information into the binary format. So Kafka consumer/producers essentially encode the schema id in schema registry with each message that is published. Then consumers can essentially pull down a message, get the schema id out, reach out to schema registry to get the proto definition, and use that definition to deserialize the message.

1 reply
So if you ETL data from Kafka to S3 dynamically, it doesn't have to know all the schemas, it can just use the schema registry to make a DynamicMessage and put it in S3 as JSON or whatever
Rohit Ramprasad
@rrampr
We are using scalapb with bazel and not sbt. Is there a way to flip on javaConversions=true with bazel
2 replies
Nimrod Sadeh
@nsadeh
Hello, I am having trouble understanding the example in https://scalapb.github.io/zio-grpc/docs/decorating. I am trying to have ZIO Logging available to my services, what would that look like? I am not sure I understand how this ZTransfrom works
Nadav Samet
@thesamet
Hi @nsadeh , I haven't tried using zio-logging, however @frekw once looked into it (https://github.com/scalapb/zio-grpc/issues/52#issuecomment-628612446) - @frekw will you be able to help out here?
3 replies