twitter/finagle

A fault tolerant, protocol-agnostic RPC system

Hamdi Allam

@hamdiallam

Have you thought about maybe using something like FailureDetector this kind of use case?

Alessandro Vermeulen

@spockz

@hamdiallam I’m not familiar with that

Hamdi Allam

@hamdiallam

It's how the ping based failure detection. In your case the ping could be checking the cert expiration, failed pings past a threshold marks the endpoint as Busy. Although if a certificate is expired, is that configured to cause a connection level failure?

Alessandro Vermeulen

@spockz

@hamdiallam the certificate example was just an example. Actually it about connect timeouts. Those count stronger than request timeouts for example

Hamdi Allam

@hamdiallam

hm, FailFast should already have an exponential backoff for connection failures. It'll mark it as busy and in the background attempt to establish a session using the exponential backoff

Alessandro Vermeulen

@spockz

Yes but still it will get a request every 5s by default to try

Hamdi Allam

@hamdiallam

how would it get a request every 5s? FailFast lives below FailureAccrual, a connection failure would cause the loadbalancer to immediately avoid the host. FailureAccrual wouldn't come into play here. The backoff is to establish that session 1s -> 32s

the FailFast backoff is not configurable. It sounds like you'd like to configure it for your usecase? Do you see the failfast/marked_dead metric increment in your service?

Alessandro Vermeulen

@spockz

So maybe the timeouts we are seeing are not triggering the fail fast to close. I’ll investigate

Orson Adams

@ParseThis

Hey Guys! super new to scala / finagle - but want to build finagle locally, any guidance on how I might get setup with that

thanks so much

sinanspd

@sinanspd

did you encounter issues when building with sbt or just not looked into that yet?

Christopher Coco

@cacoco

@ParseThis I believe we have some instructions in the CONTRIBUTING documentation that will be useful

https://github.com/twitter/finagle/blob/release/CONTRIBUTING.md

Tomasz Pasternak

@tpasternak

hey, I get this error, when I'm trying to compile develop branch

[error]     com.twitter:util-logging_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-logging_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-logging_2.12/20.9.0-SNAPSHOT/util-logging_2.12-20.9.0-SNAPSHOT.pom
[error]     com.twitter:util-stats_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-stats_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-stats_2.12/20.9.0-SNAPSHOT/util-stats_2.12-20.9.0-SNAPSHOT.pom
[error]     com.twitter:util-app_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-app_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-app_2.12/20.9.0-SNAPSHOT/util-app_2.12-20.9.0-SNAPSHOT.pom
[error]     com.twitter:util-core_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-core_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-core_2.12/20.9.0-SNAPSHOT/util-core_2.12-20.9.0-SNAPSHOT.pom
[error] (finagle-grpc-context / coursierResolutions) coursier.ResolutionException: Encountered 1 error(s) in dependency resolution:
[error]     com.twitter:util-core_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-core_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-core_2.12/20.9.0-SNAPSHOT/util-core_2.12-20.9.0-SNAPSHOT.pom

Christopher Coco

@cacoco

@tpasternak the develop branch of twitter/finagle? Have you taken a look at the CONTRIBUTING.md doc shared above (which has instructions on building the develop branch)? Any more details you can give would help to be able to figure out whats going on. Thanks!

Tomasz Pasternak

@tpasternak

oh, sorry then, I'll read it

Dermot Haughey

@hderms

anyone know of any other good Twitter research papers to read besides Your Server as a Function? Everytime I search for "twitter research paper" I seem to get legions of results for research papers that are studying some aspect of social networks

sinanspd

@sinanspd

there is a decent amount of research going on at Twitter spread throughout various domains. Do you have an area of interest or just looking for any paper that came out of Twitter ?

Dermot Haughey

@hderms

distributed systems, generally, or anything similar to Your Server as a Function

sinanspd

@sinanspd

One good starting point is to check what else that the authors of "Your Server as a Function" wrote

Distributed Systems is still quite a large bound but off the top of my head, you can check:
A Performance Modelling Service for Distributed Stream Processing Systems
The Unified Logging Infrastructure for Data Analytics at Twitter

Dermot Haughey

@hderms

great idea, thanks

sinanspd

@sinanspd

Papers on Summingbird and Algebird have some distributed components in there

Dennis

@dshkao_twitter

Hey
I've noticed that when there's a total timeout, the GlobalRequestTimeoutException does not have the remote info populated. Is this expected?

        var twitter = Http.client()
                .methodBuilder("twitter.com:80")
                .withTimeoutTotal(Duration.fromMilliseconds(110))
                .newService();
        var req = Request.apply(Method.Get(), "/");
        var res = twitter.apply(req);

com.twitter.finagle.GlobalRequestTimeoutException: exceeded 110.milliseconds to unspecified while waiting for a response for the request, including retries (if applicable). Remote Info: Not Available

Alessandro Vermeulen

@spockz

@dshkao_twitter in the case of the global timeout there is not necessarily a downstream involved or perhaps multiple. What would you have expected? Maybe the remote from the last exception of any could be useful.

Orson Adams

@ParseThis

@cacoco Hey, following up on building finagle. for context i'm super new to scala. Here is what I get when trying to build in intellj (after using dodo as instructed in CONTRIBUTING.md)

ahh ( thought I could send an image there ). The error is about not having dependencies (finding SNAPSHOTS)

not found: <location>  https://repo1.maven.org/maven2/com/twitter/util-core_2.12/20.9.0-SNAPSHOT/util-core_2.12-20.9.0-SNAPSHOT.pom
[error] (finagle-example / ssExtractDependencies) coursier.ResolutionException: Encountered 4 error(s) in dependency resolution:
[error]     com.twitter:util-logging_2.12:20.9.0-SNAPSHOT:

specific ask: how do i get these dependancies and what is the sbt build process? thanks!

Alessandro Vermeulen

@spockz

There is a dodo command for building that should setup everything correctly

Orson Adams

@ParseThis

Thanks! I ran tihis guy curl -s https://raw.githubusercontent.com/twitter/dodo/develop/bin/build | bash -s -- --no-test finagle

what should I do after that, if you dont mind

Christopher Coco

@cacoco

@ParseThis seems like your sbt isn’t trying to resolve locally built artifacts but instead is looking incorrectly at the central maven repo for snapshot releases (which is still incorrect). Firstly, you need to have the Sonatype snapshot resolver somewhere in your configuration (we recently started publishing snapshot releases but not quite ready to announce it). See: https://www.scala-sbt.org/1.x/docs/Resolvers.html#Predefined+resolvers. However, what we really want here is for sbt to resolve your locally built versions that were built via Dodo. To do that you need to remove any sbt/ivy resolution cache which for some reason may have been created and which short circuits re-resolving (though, running ./sbt update may force re-resolution but I haven’t had much luck with that). Remove ~/.ivy/cache/com.twitter directory if there is one and try again.

Alessandro Vermeulen

@spockz

Maybe the ~/.sbt/repositories file has some weird settings?

Orson Adams

@ParseThis

thanks @cacoco! That worked. Happy that I got to look deeper at sbt builds.

Alessandro Vermeulen

@spockz

Is there a way I can see in the Netty4ChannelHandler , or somewhere else, whether session resumption is used?

I suppose I should see a long handshake latency followed by very short handshake latencies if resumption is used ..

Alessandro Vermeulen

@spockz

Any reason why finagle would not resume tls sessions? We get this behaviour both with JSSE and with the boringssl bindings. The server we are connecting to supports resumption and we can connect using resume with the OpenSSL cli as well.

Alessandro Vermeulen

@spockz

Could it be that the SSLContext is created every time?

Deenar Toraskar

@deenar

Hi guys, I have having some trouble with the Finagle PostgresClient connecting to databases with SSL enabled. How do I turn up logging to look at the root cause?

I get a com.twitter.finagle.ChannelClosedException

but no more details, tried .conditionally(true, _.withTransport.tlsWithoutValidation)
.withTransport.tls("amazonaws.com")
and various combinations of these options

Ben Spencer

@dangerousben

@deenar I believe ssl is broken on the current release of finagle-postgres (finagle/finagle-postgres#195 hasn't been released)

also if you're dealing with AWS RDS, you'll have to make sure you have their ca cert in your truststore - you can get it from https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html here but it can a bit of a hassle to import - various distros have different mechanisms for achieving it

Ben Spencer

@dangerousben

unrelated question: I have an http app which occasionally spawns a background process (running on a FuturePool) that can outlive the request - what happens with tracing in this case? I assume the initial trace will be closed when the request ends - can I wrap the background process in a fresh one or something? (we use trace IDs to correlate log entries, error reports etc so it would be useful if the process had its own trace/span)

Alessandro Vermeulen

@spockz

@dangerousben , at least with the http client there is the option to just use a pem file. No need to import the rds certs anywhere

I think the span context will be preserved when you spawn a future in a future pool

As long as je initiate the future from where there is still a span context

Ben Spencer

@dangerousben

@spockz: might be possible with finagle-postgres too then, or might be easy to tweak it to do so, but I never have personally

Where communities thrive

People

Repo info

Activity