Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Yawar Quadir Amin
@yamin-oanda
hi, i'm trying to fix an issue where an http client connects to my finagle thrift service, then closes the connection abruptly. meanwhile the thrift service connects to downstream mysql, and the abrupt connection close propagates to the mysql connection and closes it. when this happens repeatedly the mysql connection pool gets churned hard and the connections get poisoned. at least this is my working theory. i saw that this cancellation propagation behaviour can be prevented using MaskCancelFilter ( https://twitter.github.io/finagle/guide/FAQ.html?highlight=mysql#what-are-cancelledrequestexception-cancelledconnectionexception-and-clientdiscardedrequestexception ), but i'm not seeing a way to use that with a Mysql RichClient as it seems to require a Service. can anyone recommend a way to make them interop?
1 reply
just fyi, this is a sample of the stack trace: ```
com.twitter.finagle.CancelledRequestException: request cancelled. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: db-sfb-rates.svc.oanda.com/172.27.44.25:3306, Downstream label: sfb_rates:read/mysql, Trace Id: 3db624dca0957e03.285dfe4f0f9aac3d<:3db624dca0957e03 2022-04-26 13:03:02,031 WARN c.t.f.u.DefaultMonitor: Exception propagated to the default monitor (upstream address: /10.42.3.78:48400, downstream address: db-sfb-rates.svc.oanda.com/172.27.44.25:3306, label: sfb_rates:read/mysql). com.twitter.finagle.mysql.PoisonConnection$PoisonedConnectionException: null 2022-04-26 13:03:02,031 WARN c.t.f.m.RollbackFactory: rollback failed when putting service back into pool, closing connection com.twitter.finagle.ChannelWriteException: com.twitter.finagle.ChannelClosedException: null at remote address: db-sfb-rates.svc.oanda.com/172.27.44.25:3306. Remote Info: Not Available from service: sfb_rates:read/mysql. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: db-sfb-rates.svc.oanda.com/172.27.44.25:3306, Downstream label: sfb_rates:read/mysql, Trace Id: 5f7d4bbea9e81ec4.4849a55aadd1cd91<:5f7d4bbea9e81ec4
Yawar Quadir Amin
@yamin-oanda
ok, figured it out. had to hand-roll the service factory to allow composing it with the MaskCancelFilter:
val mySqlClient = Mysql.client
val mask = new MaskCancelFilter[Request, Result]
val client = mySqlClient
  .withCredentials(...)
  .withDatabase(...)
  .configured(...)
val svcFactory = mask.andThen(client.newClient("host:port"))

mysql.Client(
  factory = svcFactory,
  statsReceiver = client.richClientStatsReceiver,
  supportUnsigned = client.params[UnsignedColumns].supported)
Lars
@lars-n
Hi there, I'm looking for a way to build form data http requests using finagle for http PUT (unfortunately a requirement ). There is a nice and easy way for POST, but so far I didn't see a way for any other http methods.
Alessandro Vermeulen
@spockz
@lars-n what I do is just use the methods to create the request and then I change the method to PUT on the created request before sending it to the client
Lars
@lars-n

yes, that changes the method, but unfortunately it also causes the form elements to not be encoded correctly. What would probably help is having the Netty4FormPostEncoder visible outside the package

Maybe there is a better way in a later version? We're currently on 20.12.0

Lars
@lars-n
ok, my bad. Think I figured it out. Thanks Alessandro
Yawar Quadir Amin
@yamin-oanda

hello, i'm back with another issue. i'm upgrading from Finagle 19.6.0 to 22.4.0, and a trick we were using to multiplex two separate Thrift IDLs (let's call them S1 and S2) on a single port, no longer seems to be working. our code looks like:

class CombinedService$FinagleService(
  iface: S1.MethodPerEndpoint with S2.MethodPerEndpoint,
  serverParam: RichServerParam)
  extends S1$FinagleService(iface, serverParam) {

  // We extend and load the generated S2 service so that we can expose the map of functions it provides
  private val s2 = new S2$FinagleService(iface, serverParam) {
    val functions = serviceMap
  }

  // and then add those functions to S1 so it can accept requests for S2
  for ((name, func) <- s2.functions) addService(name, func)
}

before the upgrade, this was working and both S1 and S2 methods were being served. after the upgrade (and switch from HKT-style to MethodPerEndpoint style), the S2 methods are no longer found: org.apache.thrift.TApplicationException: Invalid method name: 'getXYZ'
any ideas how to fix this?

Yawar Quadir Amin
@yamin-oanda
ok i think i figured it out. just need to create a Service manually instead of using the serveIface method to find my Thrift service:
    val thriftServer = Thrift
      .server
      // enable thrift method request counts
      .withPerEndpointStats
      .serve(serverPort(), new CombinedService$FinagleService(combinedService, RichServerParam()))
//      .serveIface(serverPort(), combinedService)
Alessandro Vermeulen
@spockz
Is anyone using dynamic time-outs with clients constructed using the method builder? The dynamic timeout isn’t picked up. Even if I replace the timeout module in the stack before instantiating the method builder
1 reply
Mike Cripps
@macripps
Does anyone have any experience sending Tracer spans to an OpenTelemetry backend? Would it just be a case of writing our own Tracer implementation like ZipkinTracer? I can't find such a thing in the community anywhere
Alessandro Vermeulen
@spockz
For OpenTracing we added our own filter. But that was mainly because of incompatibility with the trace ids format. And we wanted to add our own labels
Mike Cripps
@macripps
oh, so in your servers you just do ".filter[OpenTracingFilter]" (or equivalent) and that handles creating spans etc? That's interesting, and seems relatively easy to implement (although obviously needs a code change). Do you recall what was wrong with the trace id format?
Mike Cripps
@macripps
oh, I wonder if I could proxy through an OpenTelemetry collector which has Zipkin receiver and forward to something else, that might work "out of the box"
3 replies
Moses Nakamura
@mosesn

Does anyone have any experience sending Tracer spans to an OpenTelemetry backend? Would it just be a case of writing our own Tracer implementation like ZipkinTracer? I can't find such a thing in the community anywhere

fwiw we built a partial integration w/ opencensus (the otel predecessor) a while back. it might be of interest to you: https://github.com/twitter/finagle/blob/develop/finagle-opencensus-tracing/src/main/scala/com/twitter/finagle/tracing/opencensus/ServerTraceContextFilter.scala

8 replies
Mike Cripps
@macripps
thanks @mosesn - I think that's the approach we're going to go with rather than getting a Tracer implementation going - there's /just enough/ impedance mismatch between Finagle spans and OpenTracing spans that reusing the -zipkin-core stuff is a bit awkward. It'd be nice not to have to duplicate the Trace things but we can live that for now
Mike Cripps
@macripps
Is there any documentation for adding your own Lints to a server? I can't even find where the existing ones are defined in the code :/
1 reply
Mike Cripps
@macripps
Unrelated, has the PProf profile output format changed? I can't get the latest pprof binary to read it:
~/go/bin/pprof -http --text ~/Downloads/profile
/Users/mike/Downloads/profile: parsing profile: unrecognized profile format
pprof: failed to fetch any source profiles
4 replies
lokeshmittal10
@lokeshmittal10
Hello , i am getting the following error in the logs :
"name resolution is negative (limited dtab: Dtab() local dtab: Dtab())"
Can anyone tell how it could be remediated?
5 replies
Alessandro Vermeulen
@spockz
What was/is the rationale for allowing a (configurable) percentage of traffic that exceeds the context deadline to pass?
1 reply
Daniel Beck
@beck-daniel
Hello, I am trying to find a solution to a peculiar problem, I want to send the original serialized message bytes sent from the client down via Kafka to a second service, what would be the best way to approach thihs?
Jens
@DieBauer
Hi all, when's the next finagle release planned? last released is 22.4.0.
Mike Cripps
@macripps
Good news, Jens - 22.7.0 was released this week!
Alessandro Vermeulen
@spockz
https://finagle.github.io/blog/2022/07/28/release-notes/ also a quarterly release schedule is now used and this is the last release of the year
Alessandro Vermeulen
@spockz
What is the idea behind this new panic mode? I can see what it does. But why was it introduced? What does it fix?
Mike Cripps
@macripps
there's a bit of a description in twitter/finagle@c6060de
Panic mode is when the load balancer gives up trying to find a healthy node. The LB
sends the request to the last pick even if the node is unhealthy.
it looks like it existed before but now it's part of the API?
joybestourous
@joybestourous
hey @spockz, panic mode was introduced because it was found to reduce cascading failures, cpu throttling, etc. a variant of it had already existed under the title of "max effort". it doesn't quite fix anything, just makes our balancers more robust.
Khal!l
@redkhalil
Newbie question: How do I use a configuration (from application.conf) in HttpServer subclass? I'm able to inject configuration elsewhere, but not really sure about the HttpServer.
Denis Mikhaylov
@notxcain
Hey folks! I don't know why but we noticed that PrepareCache in finagle-mysql does not send enough CloseRequests to match PrepareRequests when the cache capacity gets reached, that leads to a growing number of prepared statements on server. We noticed that it lags behind at around 2 rps and this is the exact rate we see on the server. Any ideas on why could it happen?
Denis Mikhaylov
@notxcain
However we see that only in production. I can't reproduce that locally.
Alexey Shcherbakov
@fuCtor
Hi all! I have Finagle server and business logic on CE3 + IO, monad transformer same as in Finch. In production (highload + sometimes request cancellation) I have pending metric of server leaking. If disabling cancellation of IO monad, it's all okay. Maybe somebody saw same behaviour?
huangh
@hsl4125
Can Finagle communicate full-duplex? For example, the server pushes messages actively to the client.
ElnazAbdollahi
@ElnazAbdollahi
Hi everyone
I am a newbie. I want to check how servers are distributed simultaneously in finagle. What should I focus on?
Johnny Xie
@ardpx
Hi folks, I have a general question on offloading Netty IO thread in Finagle. Does anyone know the major difference between using the OffloadFilter/OffloadThreadPool and using the BridgedThreadPoolScheduler? Which one is recommended?
Jens
@DieBauer
How can I install a custom Monitor for the ChannelStatsHandler? No matter what I try (Client.withMonitor, or Monitor.using(monitor, service(request))), the check Monitor.isActive is always false in that Thread, which means that the full stacktrace is logged.
[agle/netty4-2-2] c.t.f.n.channel.ChannelStatsHandler$     : ChannelStatsHandler caught an exception
Rahul Bhonsale
@rahulahoop
What’s gonna happen when and if Twitter collapses ? How should transfer of maintenance be decided ? Worried but interested SoundCloud employee here 🫡
Binuh Yun
@BinuhY_twitter
Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you Fuck you
Rahul Bhonsale
@rahulahoop
Oh right
Mike Cripps
@macripps
+1, @rahulahoop
Rahul Bhonsale
@rahulahoop
+1 to fuck me or to my question? lol
Mike Cripps
@macripps
Haha, your question!
CharlieQW
@CharlieQW
Is it typical for an application (in a mostly microservices architecture) that is using finagle to have one big Service[-Req, +Rep] that leafs out to all the operations or to declare a number of Service[-Req, +Rep] on startup with a subset of operations. I think probably just having one Service[-Req, +Rep] is better but wanted to check just in case.
Alexey Shcherbakov
@fuCtor
Hi all! Somebody are using dtab? In manual I read that it support fallback if exist alternative, but I can't reproduce this behaviour. Maybe somebody can share link with example?
Mike Cripps
@macripps
hey Alexey, I've just been playing with this. It works like this: from the "top" of the dtab it keeps replacing prefixes until it reaches something with a "system" path (/$/) then it tries to resolve it. If that returns an Addr.Neg (and I think only in that case) it recurses through the tree and tries other replacements. I recently found that our SRV dns resolver was happily returning "empty list of address is okay" and that was preventing recursion
Alexey Shcherbakov
@fuCtor
Thnx, @macripps, I try again reproduce this. Maybe my custom resolver did something wrong )
Alexey Shcherbakov
@fuCtor
I found where is was mistake. It was incorrect namer implementation.
Mike Cripps
@macripps
great :) (well, you know what I mean)