by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Hamdi Allam
@hamdiallam
Have you thought about maybe using something like FailureDetector this kind of use case?
Alessandro Vermeulen
@spockz
@hamdiallam I’m not familiar with that
Hamdi Allam
@hamdiallam
It's how the ping based failure detection. In your case the ping could be checking the cert expiration, failed pings past a threshold marks the endpoint as Busy. Although if a certificate is expired, is that configured to cause a connection level failure?
Alessandro Vermeulen
@spockz
@hamdiallam the certificate example was just an example. Actually it about connect timeouts. Those count stronger than request timeouts for example
Hamdi Allam
@hamdiallam
hm, FailFast should already have an exponential backoff for connection failures. It'll mark it as busy and in the background attempt to establish a session using the exponential backoff
Alessandro Vermeulen
@spockz
Yes but still it will get a request every 5s by default to try
Hamdi Allam
@hamdiallam
how would it get a request every 5s? FailFast lives below FailureAccrual, a connection failure would cause the loadbalancer to immediately avoid the host. FailureAccrual wouldn't come into play here. The backoff is to establish that session 1s -> 32s
the FailFast backoff is not configurable. It sounds like you'd like to configure it for your usecase? Do you see the failfast/marked_dead metric increment in your service?
Alessandro Vermeulen
@spockz
So maybe the timeouts we are seeing are not triggering the fail fast to close. I’ll investigate
Orson Adams
@ParseThis
Hey Guys! super new to scala / finagle - but want to build finagle locally, any guidance on how I might get setup with that
thanks so much
sinanspd
@sinanspd
did you encounter issues when building with sbt or just not looked into that yet?
Christopher Coco
@cacoco
@ParseThis I believe we have some instructions in the CONTRIBUTING documentation that will be useful
Tomasz Pasternak
@tpasternak
hey, I get this error, when I'm trying to compile develop branch
[error]     com.twitter:util-logging_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-logging_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-logging_2.12/20.9.0-SNAPSHOT/util-logging_2.12-20.9.0-SNAPSHOT.pom
[error]     com.twitter:util-stats_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-stats_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-stats_2.12/20.9.0-SNAPSHOT/util-stats_2.12-20.9.0-SNAPSHOT.pom
[error]     com.twitter:util-app_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-app_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-app_2.12/20.9.0-SNAPSHOT/util-app_2.12-20.9.0-SNAPSHOT.pom
[error]     com.twitter:util-core_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-core_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-core_2.12/20.9.0-SNAPSHOT/util-core_2.12-20.9.0-SNAPSHOT.pom
[error] (finagle-grpc-context / coursierResolutions) coursier.ResolutionException: Encountered 1 error(s) in dependency resolution:
[error]     com.twitter:util-core_2.12:20.9.0-SNAPSHOT:
[error]         not found:
[error]             /home/tpasternak/.ivy2/local/com.twitter/util-core_2.12/20.9.0-SNAPSHOT/ivys/ivy.xml
[error]             https://repo1.maven.org/maven2/com/twitter/util-core_2.12/20.9.0-SNAPSHOT/util-core_2.12-20.9.0-SNAPSHOT.pom
Christopher Coco
@cacoco
@tpasternak the develop branch of twitter/finagle? Have you taken a look at the CONTRIBUTING.md doc shared above (which has instructions on building the develop branch)? Any more details you can give would help to be able to figure out whats going on. Thanks!
Tomasz Pasternak
@tpasternak
oh, sorry then, I'll read it
Dermot Haughey
@hderms
anyone know of any other good Twitter research papers to read besides Your Server as a Function? Everytime I search for "twitter research paper" I seem to get legions of results for research papers that are studying some aspect of social networks
sinanspd
@sinanspd
there is a decent amount of research going on at Twitter spread throughout various domains. Do you have an area of interest or just looking for any paper that came out of Twitter ?
Dermot Haughey
@hderms
distributed systems, generally, or anything similar to Your Server as a Function
sinanspd
@sinanspd
One good starting point is to check what else that the authors of "Your Server as a Function" wrote
Distributed Systems is still quite a large bound but off the top of my head, you can check:
A Performance Modelling Service for Distributed Stream Processing Systems
The Unified Logging Infrastructure for Data Analytics at Twitter
Dermot Haughey
@hderms
great idea, thanks
sinanspd
@sinanspd
Papers on Summingbird and Algebird have some distributed components in there
Dennis
@dshkao_twitter
Hey
I've noticed that when there's a total timeout, the GlobalRequestTimeoutException does not have the remote info populated. Is this expected?
        var twitter = Http.client()
                .methodBuilder("twitter.com:80")
                .withTimeoutTotal(Duration.fromMilliseconds(110))
                .newService();
        var req = Request.apply(Method.Get(), "/");
        var res = twitter.apply(req);
com.twitter.finagle.GlobalRequestTimeoutException: exceeded 110.milliseconds to unspecified while waiting for a response for the request, including retries (if applicable). Remote Info: Not Available
Alessandro Vermeulen
@spockz
@dshkao_twitter in the case of the global timeout there is not necessarily a downstream involved or perhaps multiple. What would you have expected? Maybe the remote from the last exception of any could be useful.
Orson Adams
@ParseThis
@cacoco Hey, following up on building finagle. for context i'm super new to scala. Here is what I get when trying to build in intellj (after using dodo as instructed in CONTRIBUTING.md)
ahh ( thought I could send an image there ). The error is about not having dependencies (finding SNAPSHOTS)
not found: <location> https://repo1.maven.org/maven2/com/twitter/util-core_2.12/20.9.0-SNAPSHOT/util-core_2.12-20.9.0-SNAPSHOT.pom [error] (finagle-example / ssExtractDependencies) coursier.ResolutionException: Encountered 4 error(s) in dependency resolution: [error] com.twitter:util-logging_2.12:20.9.0-SNAPSHOT:
specific ask: how do i get these dependancies and what is the sbt build process? thanks!
Alessandro Vermeulen
@spockz
There is a dodo command for building that should setup everything correctly
Orson Adams
@ParseThis
Thanks! I ran tihis guy curl -s https://raw.githubusercontent.com/twitter/dodo/develop/bin/build | bash -s -- --no-test finagle
what should I do after that, if you dont mind
Christopher Coco
@cacoco
@ParseThis seems like your sbt isn’t trying to resolve locally built artifacts but instead is looking incorrectly at the central maven repo for snapshot releases (which is still incorrect). Firstly, you need to have the Sonatype snapshot resolver somewhere in your configuration (we recently started publishing snapshot releases but not quite ready to announce it). See: https://www.scala-sbt.org/1.x/docs/Resolvers.html#Predefined+resolvers. However, what we really want here is for sbt to resolve your locally built versions that were built via Dodo. To do that you need to remove any sbt/ivy resolution cache which for some reason may have been created and which short circuits re-resolving (though, running ./sbt update may force re-resolution but I haven’t had much luck with that). Remove ~/.ivy/cache/com.twitter directory if there is one and try again.
Alessandro Vermeulen
@spockz
Maybe the ~/.sbt/repositories file has some weird settings?
Orson Adams
@ParseThis
thanks @cacoco! That worked. Happy that I got to look deeper at sbt builds.
Alessandro Vermeulen
@spockz
Is there a way I can see in the Netty4ChannelHandler , or somewhere else, whether session resumption is used?
I suppose I should see a long handshake latency followed by very short handshake latencies if resumption is used ..
Alessandro Vermeulen
@spockz
Any reason why finagle would not resume tls sessions? We get this behaviour both with JSSE and with the boringssl bindings. The server we are connecting to supports resumption and we can connect using resume with the OpenSSL cli as well.
Alessandro Vermeulen
@spockz
Could it be that the SSLContext is created every time?
Deenar Toraskar
@deenar
Hi guys, I have having some trouble with the Finagle PostgresClient connecting to databases with SSL enabled. How do I turn up logging to look at the root cause?
I get a com.twitter.finagle.ChannelClosedException
but no more details, tried .conditionally(true, _.withTransport.tlsWithoutValidation)
.withTransport.tls("amazonaws.com")
and various combinations of these options
Ben Spencer
@dangerousben
@deenar I believe ssl is broken on the current release of finagle-postgres (finagle/finagle-postgres#195 hasn't been released)
also if you're dealing with AWS RDS, you'll have to make sure you have their ca cert in your truststore - you can get it from https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html here but it can a bit of a hassle to import - various distros have different mechanisms for achieving it
Ben Spencer
@dangerousben
unrelated question: I have an http app which occasionally spawns a background process (running on a FuturePool) that can outlive the request - what happens with tracing in this case? I assume the initial trace will be closed when the request ends - can I wrap the background process in a fresh one or something? (we use trace IDs to correlate log entries, error reports etc so it would be useful if the process had its own trace/span)
Alessandro Vermeulen
@spockz
@dangerousben , at least with the http client there is the option to just use a pem file. No need to import the rds certs anywhere
I think the span context will be preserved when you spawn a future in a future pool
As long as je initiate the future from where there is still a span context
Ben Spencer
@dangerousben
@spockz: might be possible with finagle-postgres too then, or might be easy to tweak it to do so, but I never have personally