Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 24 17:43

    thesamet on master

    Update sbt-release to 1.0.13 (#… (compare)

  • Jan 24 17:43
    thesamet closed #735
  • Jan 24 17:41
    thesamet closed #727
  • Jan 24 17:40
    thesamet closed #731
  • Jan 24 17:40
    thesamet commented #731
  • Jan 24 17:38
    thesamet commented #739
  • Jan 24 11:17
    ugeshgupta000 opened #739
  • Jan 24 06:32

    thesamet on master

    Update sbt-scalajs to 0.6.32 Merge pull request #7 from xuwe… (compare)

  • Jan 24 06:32
    thesamet closed #7
  • Jan 24 06:10
    xuwei-k opened #7
  • Jan 23 20:56

    thesamet on master

    Update sbt-scalajs to 0.6.32 (#… (compare)

  • Jan 23 20:56
    thesamet closed #738
  • Jan 23 20:34

    thesamet on master

    Update sbt-scalajs to 0.6.32 Merge pull request #45 from sca… (compare)

  • Jan 23 20:34
    thesamet closed #45
  • Jan 23 20:12
    scala-steward opened #738
  • Jan 23 20:12
    scala-steward opened #45
  • Jan 23 18:25
    Charlyzzz commented #737
  • Jan 22 18:15
    thesamet commented #679
  • Jan 22 08:26
    bjaglin commented #679
  • Jan 21 20:37
    thesamet commented #56
Nadav Samet
@thesamet
@eugeneatnezasa Methods like fromName are used when you are receiving data from external sources. For example, the JSON format encodes enums as strings and uses fromName to get back to the enum. The JSON encoding is designed to work with the standard JSON format for protobufs so it isn't using scala_name. Similar argument for SparkSQL and other places where the enum is encoded/decoded from string and interoperaribility is expected.
Happy to accept a PR for scalapb/ScalaPB#353 though
@Muhammad-Bilal-Gits looks like you are using an old version of sparksql-scalapb. We no longer use UDTs.
Muhammad Bilal Shafqat
@Muhammad-Bilal-Gits
hi @thesamet
I downloaded a sample project from scalaPB github: https://github.com/thesamet/sparksql-scalapb-test. Can you point me to the latest one available?
Eugene Zhoga
@eugeneatnezasa

Happy to accept a PR for scalapb/ScalaPB#353 though

Which branch I need to use as target for PR? master? Or 0.9.x?

Eugene Zhoga
@eugeneatnezasa
Well, i created to 0.9.x
Nadav Samet
@thesamet
@Muhammad-Bilal-Gits I'vejust updated the test project to sparksql-scalapb 0.9.0. Please take a look. Don't forget to run sbt clean to have the sources regenerated.
Muhammad Bilal Shafqat
@Muhammad-Bilal-Gits
@thesamet Thanks, Now I was able to successfully make a JAR and its working in databricks. Appreciated.
Yuriy Netesov
@dehun

Hi. I am trying to make python talk to scala over grpc and encountering some issues with generating python grpc code.
I have tried doing it like

libraryDependencies ++= Seq(
      scalapbRuntime,
      scalapbRuntimeGrpc
    ),
    Compile / PB.targets := Seq(
      scalapb.gen() -> (Compile / sourceManaged).value / "protobuf",
      PB.gens.python -> (Compile / sourceManaged).value / ".."/ ".."/ ".."/ ".."/ ".."/ ".." / "old" / "src"/ "mygrpc" 
    )

And while it works well for scala - it's not generating grpc wrappers for python. Maybe somebody can point me to any example projects making this happen?
Or what will be the correct approach for this one? Producing jar with just protos and then using python's grpc_tools thingy to generate python wrappers? As far as I understood I should also include scalapb.proto into this package?

Nadav Samet
@thesamet
@dehun it looks like grpc doesn't ship a python protoc plugin. Instead they ship their own wrapper around protoc. You could try an approach where you add PB.runProtoc in Compile := (args => Process("/path/to/grpc_tools.protoc", args)!) and add to PB.targets: PB.gens.plugin("grpc_python_out") => (Compile / sourceManaged).value / "protobuf")
But it's probably better to just build the python code elsewhere (not in sbt)
you need to ship scalapb.proto only if your protos import it.
Yuriy Netesov
@dehun
thank you
Erwin Carlos Debusschere Luoni
@Charlyzzz

Hi guys!
If I want to customize a message I'll do something like this

#myFile.proto

message MyMessage {
    option (scalapb.message).companion_extends = "MyCompanionExtension";
    ...
}

Is it possible to apply that option but without the original file? (Imagine its a 3rd party definition)

Nadav Samet
@thesamet
Currently there's no way to apply options without modifying the proto file. It would be interesting to look into a way to use package scoped options (https://scalapb.github.io/customizations.html#package-scoped-options) to set options in a separate proto file. If this is interesting to you, please file a ticket on github and we can look into. Please also indicate if you're available to work on a PR to expedite it
Rôb Klér Soares da Silva Júnior
@robkler
sealed trait CurrencyType extends Snakecase

object CurrencyType extends Enum[CurrencyType] with CirceEnum[CurrencyType] with QuillEnum[CurrencyType] {

  final case object brl extends CurrencyType

  val values = findValues
}
Hello guys, I'm having problem with my akka cluster serialization.
Error:
Caused by: java.util.NoSuchElementException: is not a member of Enum (brl)
my proto:
string currency = 6 [(scalapb.field).type = "br.com.iupay.contract.model.enumer.CurrencyType"];
  implicit val typeMapperCurrencyType: TypeMapper[String, CurrencyType] =
    TypeMapper[String, CurrencyType](CurrencyType.withName)(_.entryName)
Someone have this problem?
Nadav Samet
@thesamet
Can you provide more information? What is the stack trace?
Rôb Klér Soares da Silva Júnior
@robkler
[ERROR] [2020-01-17 15:41:01.319] [Encoder]: Failed to serialize message [akka.cluster.sharding.typed.ShardingEnvelope].
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at akka.remote.serialization.ProtobufSerializer.toBinary(ProtobufSerializer.scala:109)
at akka.remote.serialization.WrappedPayloadSupport.payloadBuilder(WrappedPayloadSupport.scala:51)
at akka.cluster.sharding.typed.internal.ShardingSerializer.toBinary(ShardingSerializer.scala:37)
at akka.remote.MessageSerializer$.serializeForArtery(MessageSerializer.scala:89)
at akka.remote.artery.Encoder
KaTeX parse error: Can't use function '$' in math mode at position 5: anon$̲1.onPush(Codecs…: anon$1.onPush(Codecs.scala:139)
    at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
    at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)
    at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:624)
    at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:501)
    at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:599)
    at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter
processEvent(ActorGraphInterpreter.scala:768)
at akka.stream.impl.fusing.ActorGraphInterpreter
KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲receive$1.apply…: anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:783)
    at akka.actor.Actor.aroundReceive(Actor.scala:533)
    at akka.actor.Actor.aroundReceive$(Actor.scala:531)
    at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:690)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:573)
    at akka.actor.ActorCell.invoke(ActorCell.scala:543)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:269)
    at akka.dispatch.Mailbox.run(Mailbox.scala:230)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:242)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.util.NoSuchElementException:  is not a member of Enum (brl)
    at enumeratum.Enum.$anonfun$withName$1(Enum.scala:81)
    at scala.Option.getOrElse(Option.scala:189)
    at enumeratum.Enum.withName(Enum.scala:81)
    at enumeratum.Enum.withName$(Enum.scala:80)
    at br.com.iupay.contract.model.enumer.CurrencyType$.withName(CurrencyType.scala:8)
    at br.com.iupay.contract.protobuf.TypeMappers$.$anonfun$typeMapperCurrencyType$1(TypeMappers.scala:43)
    at scalapb.TypeMapper
anon$1.toCustom(TypeMapper.scala:26)
at com.iupay.protobuf.EscrowState$.defaultInstance$lzycompute(EscrowState.scala:369)
at com.iupay.protobuf.EscrowState$.defaultInstance(EscrowState.scala:363)
at com.iupay.protobuf.StartActor.__computeSerializedValue(StartActor.scala:21)
at com.iupay.protobuf.StartActor.serializedSize(StartActor.scala:37)
at scalapb.GeneratedMessage.toByteArray(GeneratedMessageCompanion.scala:139)
at scalapb.GeneratedMessage.toByteArray$(GeneratedMessageCompanion.scala:138)
at com.iupay.protobuf.StartActor.toByteArray(StartActor.scala:10)
Nadav Samet
@thesamet
Looks like the exception is between enumeratum and akka serialization
and not related to ScalaPB
It is possible that you are trying in the typemapper, the call to withName passed an unexpected value so you get this inner exception in enumeratum. Your proto may have a string with a value that the enum doesn't support.
Rôb Klér Soares da Silva Júnior
@robkler
Yes, but i try create my own serializer and got the same error
override def toBinary(o: AnyRef): Array[Byte] = o match {
// case e: StartActor => e.toByteArray
case e: scalapb.GeneratedMessage => e.toByteArray
case _ => throw new IllegalArgumentException("Need a subclass of scalapb.GeneratedMessage")
}
image.png
image.png
Nadav Samet
@thesamet
The exception is from within enumeration withName
Try to find out what value the typemapper is passing to it. If the value is expected, you should figure out a way how to create a typemapper instance that handles that value
Nadav Samet
@thesamet
Without throwing an exception
Rôb Klér Soares da Silva Júnior
@robkler
Ty, i will try that
Rôb Klér Soares da Silva Júnior
@robkler
message StartActor {
    option (scalapb.message).extends = "actor.BalanceActor.BalanceCommand";
    EscrowState escrowState = 1 [(scalapb.field).no_box = true];
    string ref = 2;
}
I have that proto, and if i run this code i receive the same error:
  val startState = EscrowState(
    0,
    "test",
    "PSPType.acquirer",
    0,
    0,
    CurrencyType.Brl,
    EscrowType.Authorization,
    0,
    Instant.now(),
    Instant.now(),
    "BR"
  )
  StartActor(startState, "aaa").toByteArray
That happen because the call the typer mapper from string to Currency and I need the opposite. Obs: When the call the typer mapper it pass "" as argument
Nadav Samet
@thesamet
@robkler - Looks like you are saying that the wrong direction of the typemapper is being called. Can you put together on github a minimal project that demonstrates this problem so I can try to reproduce it? You can start with sbt new scalapb/scalapb-template.g8 to get a skeleton project with ScalaPB created.
Rôb Klér Soares da Silva Júnior
@robkler
ok
Rôb Klér Soares da Silva Júnior
@robkler
Nadav Samet
@thesamet
@robkler thanks for providing this example. That was useful for helping me see what's going on. Every message in protocol buffers has a defaultInstance member where all the fields are unset/empty. This is part of the GeneratedMessage interface and is also needed by some of the internals of ScalaPB. The constraint we have is that every field needs to have a default value, including those with custom types. When ScalaPB builds up the default instance of ProtoTest, it would try to map the empty string through the type mapper and that leads to the exception you are seeing. To resolve this, you have a few options:
  1. In CurrencyType, add a case object named Unknown and map the empty string to it through your typemapper.
  2. In CurrencyType, add a case object named Unknown and override its entryName to the empty string (https://github.com/lloydmeta/enumeratum#manual-override-of-name).
  3. Instead of using string for your enum, use google.protobuf.StringValue (from wrappers.proto, see Primitive Wrappers in the docs), which is represented as Option[String] - so the default value would be None.
Rôb Klér Soares da Silva Júnior
@robkler
@thesamet Ty I fixed here =D
sosnovskiyka
@sosnovskiyka_gitlab
Hello! I have a task to write a load test on gatling.
I need to create a bi-directional grpc request. The problem is that I cannot find a good example of how to create such a query. Does anyone have any experience creating such queries? thanks
sosnovskiyka
@sosnovskiyka_gitlab
There is a speech recognition service, the request should send the audio file to the server in small parts, in response the server gives the recognized parts of the audio as text
Nadav Samet
@thesamet
@sosnovskiyka When it's bi-directional, the generated API is the same as Java. Your handler gets a StreamObserver to emit the response, and you need to return a RequestObserver that would be called back with the request events. You can start by following the Java tutorial at https://grpc.io/docs/tutorials/basic/java/ - look for "Bidirectional streaming RPC"
ugeshgupta000
@ugeshgupta000
Hi Team
I have a maven project, which i use to build till now on windows 7.
I recently switched to windows 10 and started receiving below error
--scalapb_out: protoc-gen-scalapb: This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.
Does anyone know what i can do to get this working??
Nadav Samet
@thesamet
How do you invoke ScalaPB from maven? @ugeshgupta000
There is an example here https://github.com/thesamet/scalapb-maven-example/blob/master/pom.xml - though you need to change unix:sh to windows:bat in the artifact name.