2 replies
https://github.com/twitter/finagle/blob/develop/finagle-example/src/main/scala/com/twitter/finagle/example/thrift/ThriftServer.scala
and I got this error: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". Could you give any help on how to resolve this problem ?
When upgrading to 22.1 we noticed that we are execution more retries and that our response classifier is executed more often. We get back to the old numbers when disabling H2. On a quick glance it looks like when H2 is enabled the response classifier is seeing the original errors as with H1, but now also those same wrapped in a Failure.
Does anyone have any ideas?
1 reply
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 2147483647: 2147549189 - discarded
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:503)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:489)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:376)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:419)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:332)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
1 reply
Hi, I'm creating a finagle http client with a circuit breaker like this
Http.client
.withResponseClassifier(HttpResponseClassifier.ServerErrorsAsFailures)
.withSessionQualifier.noFailFast
.withSessionQualifier.consecutiveFailuresFailureAccrual(5)
.newService("localhost:8080")And I use this client to send requests every second. The server on port 8080 is configured to always respond with 502, so I'm expecting this client to stop sending requests after 5 attempts. I see marking connection to "localhost:8080" as dead ... after 5 attempts in the logs, but the server keeps receiving requests. As far as I know, this is how Finagle's load balancer works by default: if all hosts are dead, it will still pick one randomly to send the request. But the behaviour I expect from a circuit breaker is to fail the request immediately if the breaker is open. Previously I could configure this behaviour by adding
.configured(WhenNoNodesOpenParam(WhenNoNodesOpen.FailFast))to the client. But this was removed in finagle 22.3.0: twitter/finagle@c9c21bf
Is there another way to configure a circuit breaker on a finagle http client that will stop the requests to a broken server?
1 reply
1 reply
com.twitter.finagle.CancelledRequestException: request cancelled. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: db-sfb-rates.svc.oanda.com/172.27.44.25:3306, Downstream label: sfb_rates:read/mysql, Trace Id: 3db624dca0957e03.285dfe4f0f9aac3d<:3db624dca0957e03
2022-04-26 13:03:02,031 WARN c.t.f.u.DefaultMonitor: Exception propagated to the default monitor (upstream address: /10.42.3.78:48400, downstream address: db-sfb-rates.svc.oanda.com/172.27.44.25:3306, label: sfb_rates:read/mysql).
com.twitter.finagle.mysql.PoisonConnection$PoisonedConnectionException: null
2022-04-26 13:03:02,031 WARN c.t.f.m.RollbackFactory: rollback failed when putting service back into pool, closing connection
com.twitter.finagle.ChannelWriteException: com.twitter.finagle.ChannelClosedException: null at remote address: db-sfb-rates.svc.oanda.com/172.27.44.25:3306. Remote Info: Not Available from service: sfb_rates:read/mysql. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: db-sfb-rates.svc.oanda.com/172.27.44.25:3306, Downstream label: sfb_rates:read/mysql, Trace Id: 5f7d4bbea9e81ec4.4849a55aadd1cd91<:5f7d4bbea9e81ec4
MaskCancelFilter:val mySqlClient = Mysql.client
val mask = new MaskCancelFilter[Request, Result]
val client = mySqlClient
.withCredentials(...)
.withDatabase(...)
.configured(...)
val svcFactory = mask.andThen(client.newClient("host:port"))
mysql.Client(
factory = svcFactory,
statsReceiver = client.richClientStatsReceiver,
supportUnsigned = client.params[UnsignedColumns].supported)
hello, i'm back with another issue. i'm upgrading from Finagle 19.6.0 to 22.4.0, and a trick we were using to multiplex two separate Thrift IDLs (let's call them S1 and S2) on a single port, no longer seems to be working. our code looks like:
class CombinedService$FinagleService(
iface: S1.MethodPerEndpoint with S2.MethodPerEndpoint,
serverParam: RichServerParam)
extends S1$FinagleService(iface, serverParam) {
// We extend and load the generated S2 service so that we can expose the map of functions it provides
private val s2 = new S2$FinagleService(iface, serverParam) {
val functions = serviceMap
}
// and then add those functions to S1 so it can accept requests for S2
for ((name, func) <- s2.functions) addService(name, func)
}before the upgrade, this was working and both S1 and S2 methods were being served. after the upgrade (and switch from HKT-style to MethodPerEndpoint style), the S2 methods are no longer found: org.apache.thrift.TApplicationException: Invalid method name: 'getXYZ'
any ideas how to fix this?
Service manually instead of using the serveIface method to find my Thrift service: val thriftServer = Thrift
.server
// enable thrift method request counts
.withPerEndpointStats
.serve(serverPort(), new CombinedService$FinagleService(combinedService, RichServerParam()))
// .serveIface(serverPort(), combinedService)
Does anyone have any experience sending Tracer spans to an OpenTelemetry backend? Would it just be a case of writing our own Tracer implementation like ZipkinTracer? I can't find such a thing in the community anywhere
fwiw we built a partial integration w/ opencensus (the otel predecessor) a while back. it might be of interest to you: https://github.com/twitter/finagle/blob/develop/finagle-opencensus-tracing/src/main/scala/com/twitter/finagle/tracing/opencensus/ServerTraceContextFilter.scala
8 replies
~/go/bin/pprof -http --text ~/Downloads/profile
/Users/mike/Downloads/profile: parsing profile: unrecognized profile format
pprof: failed to fetch any source profiles
Panic mode is when the load balancer gives up trying to find a healthy node. The LB
sends the request to the last pick even if the node is unhealthy.
PrepareCache in finagle-mysql does not send enough CloseRequests to match PrepareRequests when the cache capacity gets reached, that leads to a growing number of prepared statements on server. We noticed that it lags behind at around 2 rps and this is the exact rate we see on the server. Any ideas on why could it happen?
