These are chat archives for spring-cloud/spring-cloud

21st
Sep 2015
Jon McEwen
@jonmcewen
Sep 21 2015 08:28
@dsyer is RabbitMQ the go-to default right now?
Dave Syer
@dsyer
Sep 21 2015 09:23
For what?
Jon McEwen
@jonmcewen
Sep 21 2015 11:29
Seems like it's the default for spring-cloud-bus, and I just wondered why RabbitMQ over e.g. ActiveMQ
Dave Syer
@dsyer
Sep 21 2015 15:08
More lightweight, easier to install and manage
Anyway in 1.1 the bus builds on spring-cloud-stream so you get a choice
(We always intended to support other transports)
Jon McEwen
@jonmcewen
Sep 21 2015 15:20
@dsyer thanks Dave
Mark Paluch
@mp911de
Sep 21 2015 17:22
hi there
Spencer Gibb
@spencergibb
Sep 21 2015 17:24
hi
Mark Paluch
@mp911de
Sep 21 2015 17:25
wanted to chat with you about two features about tracing and correlation of distributed calls
not sure whether they are (partially available)
Spencer Gibb
@spencergibb
Sep 21 2015 17:25
ok
Mark Paluch
@mp911de
Sep 21 2015 17:26
first is, I created a log-based thingy that is similar to the zipkin traces but without the spans
so basically when a request started on one host and that request included some other remote calls, a requestId was generated in the first place and put to MDC/HTTP Request headers
so one could trace what happened during one call
the context was log related, not performance-related
sleuth has a great infrastructure for that
Spencer Gibb
@spencergibb
Sep 21 2015 17:28
we have a log implementation that was donated by 4finance
Mark Paluch
@mp911de
Sep 21 2015 17:29
not sure whether that's the same. Mine was related to log statements, that were produced during the request
I saw something about MDC, so I guess, there should be something that is similar
Do you put the traceId/spanId also to MDC?
Spencer Gibb
@spencergibb
Sep 21 2015 17:29
yes
Mark Paluch
@mp911de
Sep 21 2015 17:29
+1
these can be easily picked up by JSON/GELF loggers and then put to logstash/ElasticSearch
how about extensibility of tracing attributes?
Say, if I have a custom attribute (sessionId, processId, ...), can I pass/track that as well through the systems with sleuth?
Meaning, reading e. g. incoming HTTP headers and passing the values as outgoing HTTP headers in a transparent (invisible) style like it's done for the zipkin integration?
Spencer Gibb
@spencergibb
Sep 21 2015 17:33
yes, see spring-cloud/spring-cloud-sleuth#11
ah, you added more before I pasted that
Mark Paluch
@mp911de
Sep 21 2015 17:34
np
Spencer Gibb
@spencergibb
Sep 21 2015 17:34
there’s no facility for passing a custom attribute through all of the instrumentation.
Mark Paluch
@mp911de
Sep 21 2015 17:35
but that basically meets (and captures way more than I captured) the thing I used to build
Spencer Gibb
@spencergibb
Sep 21 2015 17:35
that sounds like a good enhancement request, though
Mark Paluch
@mp911de
Sep 21 2015 17:36
I did correlation on user/sessionId/requestId level to see what happened on different levels
since most services tend to be stateless they might not need the sessionId thing but perhaps different attributes
Adrian Cole
@adriancole
Sep 21 2015 17:36
@mp911de are you saying something like Trace.record...?
where you are choosing something not captured by default?
Mark Paluch
@mp911de
Sep 21 2015 17:37
so being able to add interceptors on the front and the back side that just capture attributes/add attributes to outgoing requests would be cool
Adrian Cole
@adriancole
Sep 21 2015 17:38
so to be clear are you saying that you want these to be in the trace's or literally you want to propagate headers to the next hop
Mark Paluch
@mp911de
Sep 21 2015 17:38
@adriancole it's more the thing of creating some sort of additional correlationId (custom) and passing that throughout all the calls
yep
Spencer Gibb
@spencergibb
Sep 21 2015 17:38
the latter was my understanding
Adrian Cole
@adriancole
Sep 21 2015 17:38
so basically you want an alternative to using trace/span attributes for correlation
Mark Paluch
@mp911de
Sep 21 2015 17:38
you would not like to propagate some Accept or Cookies headers (spoken in terms of HTTP)
I'd not see that as alternative
rather as addition
Adrian Cole
@adriancole
Sep 21 2015 17:39
well passing headers is not tied to sleuth or zipkin, right?
Mark Paluch
@mp911de
Sep 21 2015 17:39
the use case for that would be central logging
adriancole @adriancole scrolls up
Mark Paluch
@mp911de
Sep 21 2015 17:40
putting stuff to the ELK stack and so on (maybe others), but once you have the data within the trace infrastructure it's easy to use it for something
Adrian Cole
@adriancole
Sep 21 2015 17:41
@spencergibb in your demo you showed logging the trace id header for correlation reasons, right?
was that actually sleuth-specific, or generic for any header?
Mark Paluch
@mp911de
Sep 21 2015 17:43
will be back in 1hr
Mark Paluch
@mp911de
Sep 21 2015 18:03
(had to put my son into bed and it went faster than expected)
Spencer Gibb
@spencergibb
Sep 21 2015 18:04
@adriancole yup, we do log traceid
Mark Paluch
@mp911de
Sep 21 2015 18:07
to sum it up, adding support for custom attributes (adding these to the Trace/TraceScope for read/write) would be something worth implementing, right?
Mark Paluch
@mp911de
Sep 21 2015 18:26
There's one more thing. The common data model for tracing is great! I captured in my solution the whole request/response head and up to 1-2kb of the body. That helps a lot in error scenarios to see what's going on. Users should be able to access the data model to put some parts to MDC or similar for further investigation.
Added my comments to the doc
It would be amazing to use that sort of capturing within integration tests. The more context data available, the faster a bug can be found.
Adrian Cole
@adriancole
Sep 21 2015 18:41
@mp911de logging custom attributes (similar to zipkin/dapper annotations) sounds like a normal thing
but you also said reading.
what do you mean by that?
Mark Paluch
@mp911de
Sep 21 2015 18:42
the log system/MDC is not the only thing you want to do something with custom attributes
I, as developer, like to have access to the attributes for reusing them and write my own integrations
Adrian Cole
@adriancole
Sep 21 2015 18:44
so you seem to be looking for a general purpose state store
Mark Paluch
@mp911de
Sep 21 2015 18:44
Push them to graphite, create an own interceptor that pushes data to Redis, ...
yes
Adrian Cole
@adriancole
Sep 21 2015 18:44
here's some thoughts :P
Mark Paluch
@mp911de
Sep 21 2015 18:44
and your tracing infrastructure is way better than using MDC for transport
Adrian Cole
@adriancole
Sep 21 2015 18:44
firstly this came up before
as in portable JVM workshop
the trick is that this is not hard to accomplish within a specific framework
ex. bounded to sleuth, but not inheriting or coordinating with other's state (like finagle or brave)
because thread-propagation etc makes this very complicated
Mark Paluch
@mp911de
Sep 21 2015 18:45
:)
Adrian Cole
@adriancole
Sep 21 2015 18:46
ex netflix apparantly have a context propagation api internally
Mark Paluch
@mp911de
Sep 21 2015 18:46
yup, the thread-propagation (especially in async) is the hard part
you have as well a great infrastructure for that
Adrian Cole
@adriancole
Sep 21 2015 18:46
yeah, just coordinating it is where things get ugly fast
Mark Paluch
@mp911de
Sep 21 2015 18:46
that's greate for people, that know what they're doing
Adrian Cole
@adriancole
Sep 21 2015 18:47
ex. I was thinking about adapting brave to re-use finagle's state store
so that they don't make separate trace ids for the same thing, for example
this is the type of problem with mixed stuff in the same jvm
but back to your point, if a context propagation api was used the same way within spring, etc. that's possible to make an SPI out of so you can do your thing
Mark Paluch
@mp911de
Sep 21 2015 18:49
that's the idea. since spring is the leading part (not finagle or others) the direction/ownership of data should be more clear
Adrian Cole
@adriancole
Sep 21 2015 18:49
now the thing you seem to want includes propagation out of the JVM (across http transport to another)
well it works until something happens in an unmanaged thread, but yeah :P
Mark Paluch
@mp911de
Sep 21 2015 18:50
you've got the outgoing interceptors and I would, as dev to implement that feature, write my own outgoing interceptor
unmanaged threads are the dealbreaker
and that's the boundary, where the automagic ends
which does not mean, that one could propagate somehow the Trace state to apply it on the unmanaged thread, but that's where things get weird
:)
Adrian Cole
@adriancole
Sep 21 2015 18:51
so my take is to go ahead and file an issue and/or attempt an impl, but I wouldn't necessarily couple this to tracing
except that it is convenient for it
Mark Paluch
@mp911de
Sep 21 2015 18:52
will do so
Adrian Cole
@adriancole
Sep 21 2015 18:52
gRPC for example, got the relationship right
basically tracing needs this, but yeah it is its own concern
can cite or add comments here if it helps.. the doc is just a junk bin, so you can also log your use case in actual sentences :P https://docs.google.com/document/d/10S3wGxV-s2wFeXFE5ZrPAxYUW_HVJ-kUvxW6xG6_A74/edit?usp=drive_web
Mark Paluch
@mp911de
Sep 21 2015 18:53
ok
I have a last thing, that might be also interesting for you
It's remote log capturing during integration tests (that was my use case)
So basically, when you run integration tests against one remote machine (or in a distributed system) and something screws up for instance, you have to grab/correlate logs to the particular use case to see what's up
Adrian Cole
@adriancole
Sep 21 2015 18:56
yeah very cool
Mark Paluch
@mp911de
Sep 21 2015 18:56
seeking for logs on a farm on machines, especially when running a ton of tests is a crazy job
Adrian Cole
@adriancole
Sep 21 2015 18:56
so for example, twitter itself wants to start towards this soon
Mark Paluch
@mp911de
Sep 21 2015 18:56
:)
Adrian Cole
@adriancole
Sep 21 2015 18:56
currently there's no integration between zipkin and its internal log system
I'm meeting with my replacement tomorrow :P
Mark Paluch
@mp911de
Sep 21 2015 18:57
I built something, not related to zipkin, that fit my needs
it's a client and server part where logs from the JVM are stored in a Redis DB and collected by the test after its run
the nice thing is, that the log lines are displayed within the Jenkins test view and you don't have to search for the logs anymore
Adrian Cole
@adriancole
Sep 21 2015 18:58
this is cool.. screams "put example project here" :P
Mark Paluch
@mp911de
Sep 21 2015 18:58
My code is quite ok for the cases I had but not useable for spring (not generic/abstract enough)
several projects at 1&1 and Kaufland
Adrian Cole
@adriancole
Sep 21 2015 18:59
well trick is we first have to define integration test :P
Mark Paluch
@mp911de
Sep 21 2015 18:59
yes, that's the hard part here
in this context it's a test that is executed/triggered locally/by a jenkins that accesses remote services. It's not the idea of booting a spring app and then running the tests
Mark Paluch
@mp911de
Sep 21 2015 19:06
Would love to contribute stuff back, that I made for the distributed tracking/correlation and remote log capturing, if you're interested. Just ping me then
Adrian Cole
@adriancole
Sep 21 2015 19:43
well features are easier to review than systems.. maybe starting with something small like what you need for correlation
Collin Peters
@collinpeters
Sep 21 2015 20:17
Is any one else able to use the latest Brixton.M1 artifact for spring-cloud-starter-security? i.e. 1.1.0.M1?
I can't find it anywhere on repo.spring.io
Collin Peters
@collinpeters
Sep 21 2015 20:57
eureka and netflix are there... security isn't
Dave Syer
@dsyer
Sep 21 2015 20:58
possibly it missed out
I can fix it tomorrow
Collin Peters
@collinpeters
Sep 21 2015 20:59
:+1:
Dave Syer
@dsyer
Sep 21 2015 20:59
Spencer can probably do it now (I think there's a tag in github). It must have somehow not been pushed to the remote repo
Spencer Gibb
@spencergibb
Sep 21 2015 21:00
I can have a look
Spencer Gibb
@spencergibb
Sep 21 2015 22:28
@collinpeters should be there now