Another weird thing, I have enabled trace id in the logs. I'm able to see logs like this:
[info][2020-06-11_13:34:56.636] [8cb3f9b6f92ceebe|5e401da57b753cea] c.ClassName - Log MsgWhere the format is [%traceID|%spanID].
But when I open the URL http://jaeger-host:16686/trace/8cb3f9b6f92ceebe it shows 404
1 reply
Thanks @ivantopo I'm able to verify trace propagation in logs across async boundaries. Not seeing the small individual traces(ones for outound http calls or db calls) in jaeger now (as they are now part of another parent -- the WebSocket one). Not seeing the WebSocket trace in jaeger, but that's because of sampling.
This is what I did in a nutshell (so as to help others):
- Created a context in the WebSocket actor class -- instance level
- Created a span out of this context to represent the whole WebSocket session
- In message handling, Created a child span from the session-level span, put proper tags in it and forwarded messages to worker actors.
- After the worker actor, context propagation works out of the box.
I think it's fine in 1:1 request-response cases, but when 1 request has multiple replies -- like a stream -- then you need to resort to context propagation.
Nevertheless, For me, for now, it's fine even without that. I'm creating new spans for new requests from UI, from a parent span. The child spans might close too fast -- giving incorrect information, but the parent span is there for the whole life of the socket.
implicit class KamonScheduler(scheduler: akka.actor.Scheduler) {
final def scheduleOnceWithKamon(delay: FiniteDuration, receiver: ActorRef, message: Any)(
implicit
executor: ExecutionContext,
sender: ActorRef = Actor.noSender
): Cancellable = {
val ctx = Kamon.currentContext()
scheduler.scheduleOnce(delay, new Runnable {
override def run = Kamon.runWithContext(ctx) { receiver ! message }
})
}
}
Another weird behaviour I see is incorrect ordering of spans in the UI. I do a future call and in the oncomplete I send a message to an actor. The jaeger UI is showing them in reverse order. It shows the message passing was adjusted by (-x seconds), leading to an incorrect order in the UI. And if I add x to that time, it's actually correct -- as in after the future span completion. Due to the adjustment thing it's giving incorrect behaviour.
AFAIK, UI has no way of disabling adjustment.
Hello, Kamon devs!
I'm trying to understand how Kamon generates host tag. In local application configuration file I see:
kamon {
enable = yes
environment.host = "1"In base application configuration file:
kamon {
# Set to "yes", if you want to report metrics
enable = no
environment {
service = "xxx"So, in InfluxDB for Histogram metrics made by Kamon I see:
host =1
instance=xxx@1
service=xxxBut, if an org.influxdb.Point was written directly using the org.influxdb.InfluxDB driver, the metric also contains tag host that equals to hostname of the machine. By the way, application also reports JVM metrics using kamon-system-metrics. Is it possible that Kamon that is not used to produce some metric adds its host to it?
Hello, Kamon devs
I am having some issues about using tapir with akka-http backend together with kamon? I observe problem with resolving operation names in the span metrics, and wonder if there's some workaround ? cc @matwojcik I see you have faced this issue before. Do you find a solution for the same ?
kamon.environment.host set to auto and only change it if you really need to. For example, in our own deployments we use an environment variable with the actual name of the host since we are running everything in containers. There are a few settings in the InfluxDB reporter to decide whether you want to set the host tag or not