by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Nik Gushchin
    @Dr-Nikson

    Hm, I’m trying to build my protos, but there a lots of

    object fs2 is not a member of package higherkindness.mu.rpc.internal.server
    [error]   @service(Protobuf, Identity, namespace = Some("uadc.services.v1"), methodNameStyle = Capitalize) trait SystemInstrumentsController[F[_]] {

    What am I doing wrong?)

    Nik Gushchin
    @Dr-Nikson
    Ah, nvm. I missed mu-rpc-fs2. I think it would be great to have mention of the additional deps in the tutorial
    Dermot Haughey
    @hderms
    Are there any plans to make handling forced optionality in proto 3 more idiomatic?
    Alec Zorab
    @aleczorab_twitter
    is there any support for GRPC reflection protocol? I can't find anything in the repo, but maybe someone's written a plugin for it?
    Alec Zorab
    @aleczorab_twitter
    also, if, skeuomorph doesn't seem to know about the standard protobuf types, google/protobuf/empty.proto: File not found. Do I need to go and source those myself?
    Leif Battermann
    @battermann

    Hi, we implemented a gRPC client with mu-scala which worked really well. However when running the client we always get the following error message:

    0: Data Service Response (100): --> io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdown invoked
    io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdown invoked
            at io.grpc.Status.asRuntimeException(Status.java:533)
            at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:460)
            at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
            at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
            at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
            at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
            at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
            at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
            at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
            at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

    Network diagnostics show that host and port is accessible.
    Appearently SSL/TLS is used.
    We added the certificate to the keystore as descibed here: https://www.47deg.com/blog/mu-rpc-securing-communications-with-mu/#client-6
    And we still get the same error. It is quite hard to debug this.
    Do you have any pointers what to look into? Is it possible to enable more verbose logging?

    Leif Battermann
    @battermann
    When running this keytool -import -alias sample-ssl -file certificate.pem -keystore sample-ssl-keystore I am asked for a password, an empty password is not allowed.
    How can I tell mu the password for the keystore?
    Leif Battermann
    @battermann
    I found another java option: -Djavax.net.ssl.trustStorePassword… I will try this
    Leif Battermann
    @battermann
    no luck, so far
    Leif Battermann
    @battermann
      def run(args: Args): IO[ExitCode] = {
        val channelFor: ChannelFor                       = ChannelForAddress(args.host, args.port)
        val serviceClient: Resource[IO, DataService[IO]] = DataService.client[IO](channelFor, Nil, null)
    
        for {
          response <- serviceClient.use(client => client.ListOrders(empty.Empty()))
          _        <- IO(response.take(100).dump("Data Service Response (100):").subscribe())
        } yield ExitCode.Success
      }
    I always get the same output (see above) no matter what I pass in or if the keystore exists or anything. So e.g. if I pass a wrong host or port… same output.
    How can I troubleshoot this?
    Leif Battermann
    @battermann
    using rpccurl revealed that the certificate we are using might not be correct…
    Leif Battermann
    @battermann
    I found out that the service uses basic authorization and with this header it works: -H 'Authorization: Basic Zm9vYmFyOmZvb2Jhci1wYXNz’ when using grpcurl
    So now I will try to find out how to set a custom header using the mu-scala client
    Leif Battermann
    @battermann
    looks like it can be done like this:
    case class AuthCredentials(user: String, pass: String) extends CallCredentials {
      override def applyRequestMetadata(requestInfo: RequestInfo, executor: Executor, metadataApplier: MetadataApplier): Unit =
        executor.execute(new Runnable() {
          override def run() =
            try {
              val key            = Metadata.Key.of("Authorization", io.grpc.Metadata.ASCII_STRING_MARSHALLER)
              val userPassBase64 = getMimeEncoder.encodeToString(s"$user:$pass".getBytes(StandardCharsets.UTF_8))
              val value          = s"Basic $userPassBase64"
              val headers        = new Metadata()
              headers.put(key, value)
              metadataApplier.apply(headers)
            } catch {
              case err: Throwable => metadataApplier.fail(Status.UNAUTHENTICATED.withCause(err))
            }
        })
    
      override def thisUsesUnstableApi(): Unit = ()
    }
    haven’t tested it yet
    Leif Battermann
    @battermann
    So, using grpcurl it works with this command:
    grpcurl -v \
      -H 'Authorization: Basic Zm9vYmFyOmZvb2Jhci1wYXNz' \
      -import-path protocol/src/main/resources \
      -proto protocol/src/main/resources/data_service.proto \
      -cacert cert.pem \
      host:12345 \
      my.Service/Method
    This means the issue is not on the server side. How can I implement the same query with mu-scala? I would really appreciate some pointers :pray:
    Chris Birchall
    @cb372
    @battermann wow that's quite a mouthful just to add basic auth! It's probably worth adding that helper to Mu.
    Leif Battermann
    @battermann
    Yes :) I don’t even know if it works
    it compiles and when I run it the output is always the same...
    chain-alex
    @chain-alex
    Do the generated protocol case classes have a fromBytes constructor I can use?
    specifically for proto
    Leif Battermann
    @battermann
    @cb372 we solved the problem
    part of the problem was here:
        for {
          response <- serviceClient.use(client => client.ListOrders(empty.Empty()))
          _        <- IO(response.take(100).dump("Data Service Response (100):").subscribe())
        } yield ExitCode.Success
    the client resource gets destroyed before we run the IO response...
    :see_no_evil:
    chain-alex
    @chain-alex
    I am trying to use PBDirect but it is not clear to me how to do the above
    Dylan Martin
    @dmarticus
    hi @chain-alex I’m sorry for the slow response to your question! We use our fork of PBDirect under the hood of mu-scala for Protobuf serialization, and it’s possible to define custom codecs for both Avro and protobuf serialization by using instances of PBDirect’s PBScalarValueReader and PBScalarValueWriter type classes (see here: https://higherkindness.io/mu-scala/guides/custom-grpc-serialization#custom-codecs). What specifically are you trying to do in your case? FYI, for deserialization, PBDirect supports a pbTo[A] method if that’s what you’re looking for (example: https://github.com/47degrees/pbdirect#deserialization)
    Anton Semenov
    @a7emenov

    Hi, everyone!

    Am I correct that mu only respects the package option (and requires it) when generating server/client definitions from .proto files? Some of the proto files I'm dealing with have package names that I don't want to use when generating Scala code and some have no package names at all.

    I could've manually provided the target directory for generation that points to the package I want (which would only work assuming that all generated classes should reside in the same package), but that will still fail if there is no package name in the .proto file - I always have to provide it.

    I really want to keep the original package names in the .proto definitions, because that's important when communicating with other teams, so is there a way I could "customize" the target package without having to modify the .proto files?

    Dylan Martin
    @dmarticus
    Hi @a7emenov! You're correct that mu requires the package option when generating server/client definitions from .proto files. We do have an option for customizing the target directory (to an extent) that can be configured in your build.sbt (or similar) called muSrcGenIdlTargetDir (more details can be found here: https://higherkindness.io/mu-scala/reference/source-generation). This plugin setting determines the name of the target directory to write your generated protobuf code into (the default being target/scala-2.13/src_managed/main/), so I think it would be possible to use that plugin to specify different target directories for your generated protobuf code if you don't want to modify the package names. That said, I don't know if this is a complete answer to your question, and I'm happy to help you further if you can provide more specific details to what you're trying to do :)
    Anton Semenov
    @a7emenov

    @dmarticus thank you for the response, really appreciate it! Basically confirms all my assumptions. Though I really love Mu, this may be the reason why we'd have to switch to ScalaPB as it allows to be very flexible when generating packages.

    Is it likely that in the future skeuomorph and, by extension, mu will support options like java_package?

    Dylan Martin
    @dmarticus
    @a7emenov great feedback; and I'll circle back with the mu folks at 47 to see what our roadmap for skeuomorph looks like going forward (we're planning on continuing to make mu and skeuomorph more tightly coupled, e.g. we're likely going to move from avrohugger to skeuomorph for the avro schema parsing), so no promises but I'll definitely bring it up. I've added the issue (higherkindness/skeuomorph#318) to the skeuomorph backlog if you'd like to follow along on it.
    Anton Semenov
    @a7emenov
    @dmarticus thanks again - looking forward to this being implemented.
    chain-alex
    @chain-alex
    How hard would it be to use Mu with gradle?
    chain-alex
    @chain-alex
    also, how can I convert an instance of a proto generated case class to a byte array?
    Dylan Martin
    @dmarticus
    Hi @chain-alex, does the pbTo[A] method from https://github.com/47degrees/pbdirect#deserialization not work for writing the case classes to a byte array? And re: the mu with gradle, I've never tried it before (I don't use gradle for scala projects) but I imagine the scala plugin for gradle will work with this library (https://guides.gradle.org/building-scala-libraries/). Not sure if you've tried that approach already; please let me know if you did and you're getting errors
    Andrey Patseev
    @patseev
    Hi guys. Have a question regarding gRPC mu client. What happens when connection to remote gRPC server closes? I see that the service is being exposed as a resource, does that mean that it terminates upon loss of connection and I have to reboot the app to work again? Or is the reconnection handled in the background automatically?
    Andrey Patseev
    @patseev
    Maybe I just misuse it and it brings confusion. What is the best pattern for working with it? Right now I initiate resource at a top level and then pass the client down into app wiring, then use that client on lower levels of the app. Should I instead pass the resource on the lower levels, and when I need to call the server, do it via .use(client => ...)?
    Dylan Martin
    @dmarticus

    Right now I initiate resource at a top level and then pass the client down into app wiring, then use that client on lower levels of the app. Should I instead pass the resource on the lower levels, and when I need to call the server, do it via .use(client => ...)?

    Hi Andrey, the pattern you described where you pass the resource into the lower levels and then call it via .use(client => ...) makes sense from my perspective! I don't know what specifically you're working on, so I can't offer more insights than that, but have you checked out our example repo for using mu? You can check it out here (https://github.com/higherkindness/mu-scala-examples/); this repo contains several examples that use mu idiomatically (here's one that does the behavior that you're describing: https://github.com/higherkindness/mu-scala-examples/blob/master/tracing/serverA/src/main/scala/com/example/ServerA.scala#L47).

    @patseev ^ if you didn't get notified :)
    Juan Pedro Moreno
    @juanpedromoreno
    I think you are doing right, @patseev . Passing the client down is the common pattern used.
    Dylan Martin
    @dmarticus
    hi @a7emenov, with the latest release of v0.25, skeuomorph now supports java_package options with the same functionality as ScalaPB. See the release notes for more!
    Anton Semenov
    @a7emenov
    Looking forward to respective mu release as well, thanks a lot @dmarticus !
    Dylan Martin
    @dmarticus
    Hi @a7emenov, mu version 0.23.0 and sbt-mu-srcgen (the sbt plugin that handles the source generation) version 0.23.0 have both been released as of this afternoon! The release notes can be found here for mu and here for sbt-mu-srcgen. Enjoy!
    Anton Semenov
    @a7emenov
    Didn't know it's Christmas time already, @dmarticus :D
    This means mu stays in the project, cannot underappreciate that.
    Anton Semenov
    @a7emenov

    Hey @dmarticus!

    Created higherkindness/skeuomorph#337 as there seems to be an issue with enums still requiring the package option. Really hope it can be fixed soon, but luckily it's not a complete blocker for us.

    Please let me know if I can help with it in any way.

    Dylan Martin
    @dmarticus
    Thanks for letting me know @a7emenov! I took a look at the issue and left a small comment but the issue itself seems clearly filed and make sense. This week I'm a bit tied up with consulting work during the week but I can try and take a look at this tonight or over the weekend. If one of my coworkers has more bandwidth, though, I can ask and see if they'd take a look at it. Thanks again for the detailed bug report!
    Anton Semenov
    @a7emenov
    Much obliged @dmarticus! As again, happy to help and/or provide any necessary information.