but I'm not running in dev mode as of now. It's packaged as docker image and running in k8s.
are you able to access the status page on that container and see what's the status of the instrumentation modules?
Actually I was targeting 3 of our services with Kamon. 2 of them are working great. The 3rd one had WebSockets. In that also, HTTP is working fine, only WebSocket is problematic. I've been debugging that since 2 days now, tried doing create context at different places (mostly wrapping the message passing to worker actors), but looks like somewhere it gets lost or something.
curl for instrumentation module gives this json
{
"present": true,
"modules": {
"annotation": {
"name": "Annotation Instrumentation",
"description": "Provides a set of annotations to create Spans and Metrics out of annotated methods",
"enabled": true,
"active": false
},
"akka-http": {
"name": "Akka HTTP Instrumentation",
"description": "Provides context propagation, distributed tracing and HTTP client and server metrics for Akka HTTP",
"enabled": true,
"active": true
},
"executor-service": {
"name": "Executor Service Instrumentation",
"description": "Provides automatic Context propagation to all non-JDK Runnable and Callable implementations which enables\n Context propagation on serveral situations, including Scala, Twitter and Scalaz Futures",
"enabled": true,
"active": true
},
"play-framework": {
"name": "Play Framework Instrumentation",
"description": "Provides context propagation, distributed tracing and HTTP client and server metrics for Play Framework",
"enabled": true,
"active": true
},
"mongo-driver": {
"name": "Mongo Driver Instrumentation",
"description": "Provides automatic tracing of client operations on the official Mongo driver",
"enabled": true,
"active": false
},
"akka-remote": {
"name": "Akka Remote Instrumentation",
"description": "Provides distributed Context propagation and Cluster Metrics for Akka",
"enabled": true,
"active": true
},
"jdbc": {
"name": "JDBC Instrumentation",
"description": "Provides instrumentation for JDBC statements, Slick AsyncExecutor and the Hikari connection pool",
"enabled": true,
"active": false
},
"scala-future": {
"name": "Scala Future Intrumentation",
"description": "Provides automatic context propagation to the thread executing a Scala Future's body and callbacks",
"enabled": true,
"active": true
},
"akka": {
"name": "Akka Instrumentation",
"description": "Provides metrics and message tracing for Akka Actor Systems, Actors, Routers and Dispatchers",
"enabled": true,
"active": true
},
"logback": {
"name": "Logback Instrumentation",
"description": "Provides context propagation to the MDC and on AsyncAppenders",
"enabled": true,
"active": true
}
},
"errors": {}
}
I would start logging the current trace id everywhere and see where it gets lost
if it is on a future, actor or something we already support then maybe there is an issue with the agent or initialization... if it is on something else that we don't support at the moment then new (or manual) instrumentation will be necessary
can you please try to narrow it down the specific interaction where the context is lost? and of course, ask if you need any help!
:(
those are the typical places where a context would be lost
Another weird thing, I have enabled trace id in the logs. I'm able to see logs like this:
[info][2020-06-11_13:34:56.636] [8cb3f9b6f92ceebe|5e401da57b753cea] c.ClassName - Log MsgWhere the format is [%traceID|%spanID].
But when I open the URL http://jaeger-host:16686/trace/8cb3f9b6f92ceebe it shows 404
I don't remember whether we have a conversion rule for the sampling decision yet
we should encourage users to log the sampling decision as well!
1 reply
And does sampling imply that if I try with just one websocket session, it might not even come up in jaeger? I need to try multiple sessions?
regarding sampling, yes
the first Span of the chain takes a sampling decisions and then all related spans just follow the same sampling decision
that's the way to go
Thanks @ivantopo I'm able to verify trace propagation in logs across async boundaries. Not seeing the small individual traces(ones for outound http calls or db calls) in jaeger now (as they are now part of another parent -- the WebSocket one). Not seeing the WebSocket trace in jaeger, but that's because of sampling.
This is what I did in a nutshell (so as to help others):
- Created a context in the WebSocket actor class -- instance level
- Created a span out of this context to represent the whole WebSocket session
- In message handling, Created a child span from the session-level span, put proper tags in it and forwarded messages to worker actors.
- After the worker actor, context propagation works out of the box.
@ivantopo I understand I should be able to get a view like this one from zIpkin on the APM dashboard, but I have no idea how, can you give me some insights please? https://zipkin.io/public/img/web-screenshot.png
Is it possible to be assigned to an issue @ivantopo ? This is an issue that NR can fix : kamon-io/Kamon#789. So wanted to be assigned to it so I can track and prioritize it.
I think it's fine in 1:1 request-response cases, but when 1 request has multiple replies -- like a stream -- then you need to resort to context propagation.
Nevertheless, For me, for now, it's fine even without that. I'm creating new spans for new requests from UI, from a parent span. The child spans might close too fast -- giving incorrect information, but the parent span is there for the whole life of the socket.
I had this conversation before
I know
ended up doing this
implicit class KamonScheduler(scheduler: akka.actor.Scheduler) {
final def scheduleOnceWithKamon(delay: FiniteDuration, receiver: ActorRef, message: Any)(
implicit
executor: ExecutionContext,
sender: ActorRef = Actor.noSender
): Cancellable = {
val ctx = Kamon.currentContext()
scheduler.scheduleOnce(delay, new Runnable {
override def run = Kamon.runWithContext(ctx) { receiver ! message }
})
}
}