Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
Repo info
  • Mar 23 12:54
    ivantopo commented #1094
  • Mar 23 09:34
    ivantopo commented #1094
  • Mar 23 08:43
    ivantopo commented #1094
  • Mar 23 08:42
    ivantopo commented #1144
  • Mar 23 08:36
    ivantopo opened #1147
  • Mar 23 02:34
    mofei100 commented #1144
  • Mar 23 02:33
    mofei100 closed #1144
  • Mar 21 15:36
    seglo edited #1145
  • Mar 21 15:33
    seglo synchronize #1145
  • Mar 21 08:25
    sebarys opened #1146
  • Mar 20 18:16
    sebarys commented #793
  • Mar 20 18:14
    sebarys commented #793
  • Mar 19 23:41
    seglo opened #1145
  • Mar 18 14:33
    cmcmteixeira commented #1127
  • Mar 18 02:42
    mofei100 edited #1144
  • Mar 18 02:22
    mofei100 opened #1144
  • Mar 17 19:48
    hughsimpson commented #1127
  • Mar 17 19:10
    hughsimpson commented #1127
  • Mar 17 11:55
    ivantopo closed #1046
  • Mar 17 11:55
    ivantopo commented #1046
Ivan Topolnjak
oh man
I had this conversation before
I know
Ivan Topolnjak
I don't know where did I write it down, but I remember having this conversation several times and realizing that we have to instrument all the scheduleOnce calls to keep the same context
4 replies
Rajat Khandelwal
any band-aid fix I can do? Other than have the "websocketcontext" propagate down everywhere? Problem is, it's not quite readable when some message handlers will be Kamon.runWithContext, and some won't.
Rajat Khandelwal
ended up doing this
  implicit class KamonScheduler(scheduler: akka.actor.Scheduler) {
    final def scheduleOnceWithKamon(delay: FiniteDuration, receiver: ActorRef, message: Any)(
      executor: ExecutionContext,
      sender:   ActorRef = Actor.noSender
    ): Cancellable = {
      val ctx = Kamon.currentContext()
      scheduler.scheduleOnce(delay, new Runnable {
        override def run = Kamon.runWithContext(ctx) { receiver ! message }
Franco Albornoz
Hey everybody, I'm trying to migrate an existing library that uses kamon 1.x to 2.x and am almost done, but I'm struggling with migrating some custom http clients which used the Kamon.withContextKey method, which I believe now should be replaced with preStartHooks, but can't seem to find any documentation about how to use those. Could you maybe point me in the right direction there?
Rajat Khandelwal
Hey, is there a config to disable kamon for test cases? I don't need to instrument tests.
3 replies
Rajat Khandelwal


Incorrect "invalid parent span id". Weird behaviour

Rajat Khandelwal

Another weird behaviour I see is incorrect ordering of spans in the UI. I do a future call and in the oncomplete I send a message to an actor. The jaeger UI is showing them in reverse order. It shows the message passing was adjusted by (-x seconds), leading to an incorrect order in the UI. And if I add x to that time, it's actually correct -- as in after the future span completion. Due to the adjustment thing it's giving incorrect behaviour.

AFAIK, UI has no way of disabling adjustment.

Ivan Topolnjak
hey @prongs, that issue with Jaeger.. is it just about the order in which the Spans are shown in the UI or the parent-child relationships are wrong?
Alexey Kiselev

Hello, Kamon devs!
I'm trying to understand how Kamon generates host tag. In local application configuration file I see:

kamon {
  enable = yes
  environment.host = "1"

In base application configuration file:

kamon {
  # Set to "yes", if you want to report metrics
  enable = no

  environment {
    service = "xxx"

So, in InfluxDB for Histogram metrics made by Kamon I see:

host =1

But, if an org.influxdb.Point was written directly using the org.influxdb.InfluxDB driver, the metric also contains tag host that equals to hostname of the machine. By the way, application also reports JVM metrics using kamon-system-metrics. Is it possible that Kamon that is not used to produce some metric adds its host to it?

Arjun Karnwal

Hello, Kamon devs

I am having some issues about using tapir with akka-http backend together with kamon? I observe problem with resolving operation names in the span metrics, and wonder if there's some workaround ? cc @matwojcik I see you have faced this issue before. Do you find a solution for the same ?

Ivan Topolnjak
hey @alexeykiselev, what Kamon versions are you using there? some of those names sound like from previous Kamon versions! Regarding the host tag, ideally you will leave kamon.environment.host set to auto and only change it if you really need to. For example, in our own deployments we use an environment variable with the actual name of the host since we are running everything in containers. There are a few settings in the InfluxDB reporter to decide whether you want to set the host tag or not
@arjunkarnwal what exactly is the problem you are seeing?
Arjun Karnwal
@ivantopo when I use tapir with akka-http backend together with kamon, I observer that span metrics like span_processing_time_seconds_count{operation="http.server.request"} whereas if I dont use Tapir I get span_processing_time_seconds_count{operation="api/v1/mycustomerAPIPath"} . I dont know why but when I use tapir, the operation attribute gets overridden i.e. instead of having the api path, it contain "http.server.request".
6 replies
Alexey Kiselev

hey @alexeykiselev, what Kamon versions are you using there?

It's 2.1.0, but configuration may be from older versions.
We set host to internal node ID to show Kamon histograms in Grafana the same as we show metrics created directly. In latter case we just add tag node with the same ID.
I wonder, is it possible in project where Kamon is used and kamon-system-metrics also used that direct calls to org.influxdb.InfluxDB some how polluted with tags that Kamon set.

2 replies
Red Benabas
Hi Kamon dev! We're upgrading our SBT native packager project from Kamon 1.X to Kamon 2.X, following the steps in the guide https://kamon.io/docs/latest/guides/migration/from-1.x-to-2.0/. We've added Kanela plugin as well as Kanela agent. When I point project to a zipkin sever running on localhost we can see the traces. However, not on remote zipkin server. Has anyone come across this?
Also, is there a way enable DEBUG level logging in kamon.zipkin?
7 replies
Alexis Hernandez
Has anyone else integrated kamon on a play server that uses grpc clients generated by scalapb? I'm experienced some weird issues, where some grpc calls get invoked twice with no reason, my belief is that kamon may be the cause (I'm still experimenting)
7 replies
Yaroslav Derman
Hello. Does anyone try to use kamon with zio ?
Hi! Is it possible to check connection status to metrics db with Kamon?
4 replies
Jakub Kozłowski
Hi, how could I have Kamon-reported spans include the tags from the context?
10 replies
if that's not possible, I can see I can call .tag on the span directly, but I can't read it that way - or can I?
Hi kamon aficionados, I would like to get your opinion on this subject matter. I've gotten feedback from a colleague that kamon has issues with the byte code generation, that it could damage your artifact, well in short that it's not reliable. And of course he heard this from colleagues who are not around anymore, so I can't verify this. Anybody has had such an experience before?
5 replies
Daniel van der Ende
Hi, I'm trying to integrate Kamon with an Akka HTTP app. The docs seem to suggest that it should work out of the box (e.g. add dependency to build.sbt, run Kamon.init()). I can see JVM and other system metrics, but 0 Akka HTTP metrics. Any idea what the problem here could be? Thanks!
34 replies


we use for our Scala application the Kanela agent 1.0.5. Our application runs with openjdk-14.01 in a docker container.
When the kanela agent is initialized it partially stops.

_  __                _        ______
| |/ /               | |       \ \ \ \
| ' / __ _ _ __   ___| | __ _   \ \ \ \
|  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
| . \ (_| | | | |  __/ | (_| |  / / / /
|_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/

Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.5)
[main] INFO 2020-06-29 12:45:02  Logger : The Module: Executor Service Capture on Submit Instrumentation is disabled
[main] INFO 2020-06-29 12:45:02  Logger : The Module: Akka Remote Instrumentation is disabled
[main] INFO 2020-06-29 12:45:02  Logger : Loading Akka Instrumentation
[main] INFO 2020-06-29 12:45:02  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.EnvelopeInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.SystemMessageInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.RouterInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.ActorInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.ActorLoggingInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.AskPatternInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.EventStreamInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.ActorRefInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.akka_25.DispatcherInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.akka_26.DispatcherInstrumentation
[main] INFO 2020-06-29 12:45:04  Logger : Loading Executor Service Instrumentation
[main] INFO 2020-06-29 12:45:04  Logger :  ==> Loading kamon.instrumentation.executor.ExecutorTaskInstrumentation

Without docker we don't have this problem. With openjdk-13 it also works with docker. Does anyone know this behaviour?

9 replies
Yaroslav Derman

Hello. Does anyone try to use kamon with zio ?

@ivantopo Have you tried to run kamon with zio, because I don't have context propagation and correct span creation ((

Ivan Topolnjak
no, didn't try it myself
didn't even try ZIO yet
Hi, is there any interest to allow Kamon's Context to be serialised in HTTP according to the W3C Correlation Context proposal? https://w3c.github.io/correlation-context/
I could see this as a big plus for interoperability in a mixed-tech architecture. It's not standardised yet, but once finalised, would you consider it/would you be open for a PR or does something speak against it in general?
I could see it implemented similar to Trace Context Codecs so that as a user you could choose how to serialise it.
ATM the only difference in how Kamon serialises context and the W3C proposal is that key-value pairs are separated with ; in Kamon and with , in the proposal (apart from the actual name of the header value - which is configurable already)
5 replies
Franco Albornoz
Hey guys, so I had a shared library with Kamon 1.x that used the SpanCustomizer which I am trying to replace now with a PreStartHook. The thing is I used to have the customizer with a constructor with two parameters which were used to then customize the spans. The thing is PreStartHook needs to have a parameterless constructor, but I'm not sure how I'd be able to customize the spans more dynamically with it.
6 replies
Hi all, is there a preferred way to propagate trace contexts and context tags via GRPC? I can see this issue has been around for a while kamon-io/Kamon#616 (assuming this is for akka-grpc) and there is also this repo: https://github.com/nezasa/kamon-akka-grpc. Is there a plan for the future?
4 replies

Hello, Kamon dev. Here’s my config https://controlc.com/cf392b75. I can’t send anything to influxdb with Kamon because of

ERROR kamon.influxdb.InfluxDBReporter - Metrics POST to InfluxDB failed with status code [404], response body: {"error":"database not found: \"mydb\””}

error. I can’t understand where did it find mydb? I have db0 in configs.

7 replies
Ramakrishna Hande
Hi, Is the header name X-B3-TraceId renamed to trace-id ? We were using some old version of zipkin but now after some upgrade we have response having header trace-id. We are using Zipkin version 2.12.3. Thanks in advance
3 replies
Hello, there! Is it possible to add multiple custom fields to histogram? Not the tags. And is there any way to create a custom metric with Kamon without fields p50.0 p70.0 p90.0 p95.0 p99.0 p99.9 service?
2 replies
Hi, after upgrading to bundle and prometheus 2.1.2 I'm getting the following exception upon startup
Unable to start Kanela Agent. Please remove -javaagent from your startup arguments and contact Kanela support.: java.lang.NoClassDefFoundError: com/datastax/driver/core/Session
6 replies
I don't want to ask like this but since my pull-request is being ignored completely for almost 2 months i don't know how else to contact. Is there something that i forgot to do?
17 replies
Alexis Hernandez
BTW, the gitter activity on the right hasn't been updated for a year
1 reply
Alexis Hernandez
Last item is from 2019
Ivan Topolnjak
ha, didn't see that before
Enzo Bonggio

Hi I’m using kanela-agent on a play project and I want to add support for httpok. So I added on my build.sbt -> "io.kamon" %% "kamon-okhttp" % "1.0.3” and the configuration to enable it as it says on reference.conf
The error that I’m facing right now is the following

Caused by: java.lang.ClassNotFoundException: kanela.agent.api.instrumentation.KanelaInstrumentation
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at kanela.agent.util.classloader.ChildFirstURLClassLoader.loadClass(ChildFirstURLClassLoader.java:58)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at kanela.agent.InstrumentationLoader.lambda$loadInstrumentation$b85f757d$1(InstrumentationLoader.java:58)
    at kanela.agent.libs.io.vavr.control.Try.of(Try.java:75)
    at kanela.agent.InstrumentationLoader.loadInstrumentation(InstrumentationLoader.java:56)
    at kanela.agent.InstrumentationLoader.lambda$null$1(InstrumentationLoader.java:46)
    at kanela.agent.libs.io.vavr.collection.List.flatMap(List.java:898)
    at kanela.agent.InstrumentationLoader.lambda$load$4(InstrumentationLoader.java:46)
    at kanela.agent.libs.io.vavr.collection.List.map(List.java:1038)
    at kanela.agent.InstrumentationLoader.load(InstrumentationLoader.java:43)
    at kanela.agent.Kanela.lambda$null$0(Kanela.java:79)
    at kanela.agent.util.classloader.InstrumentationClassPath.use(InstrumentationClassPath.java:84)
    at kanela.agent.Kanela.lambda$start$1(Kanela.java:68)
    at kanela.agent.util.Execution.timed(Execution.java:32)
    at kanela.agent.util.Execution.runWithTimeSpent(Execution.java:41)
    at kanela.agent.Kanela.start(Kanela.java:67)
    at kanela.agent.Kanela.premain(Kanela.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)

Should I add another dependency ? Also I could not find any examples of this so I’m lost

3 replies
Krunoslav Uzelac
Hi guys, I'm having the Unable to start Kanela Agent. Please remove -javaagent from your startup arguments and contact Kanela support issue
javaAgents += "io.kamon" % "kanela-agent" % "1.0.5",
addSbtPlugin("io.kamon" % "sbt-kanela-runner" % "2.0.6")
addSbtPlugin("com.lightbend.sbt" % "sbt-javaagent" % "0.1.5")
any hints? :)
will try 2.1.3
Ivan Topolnjak
hey @Reeebuuk, indeed, try 2.1.3
Pavel Pliachystsik
Hello everybody! I have Play framework application (2.8.1) and subproject with Akka cluster (2.6.3). I'm trying to integrate kamon, added play agent and javaagent to pluguins, enabled it in root project and see some attach errors:
[Attach Listener] ERROR 2020-07-10 09:23:15 Logger : Error => akka.dispatch.Envelope with message Cannot resolve type description for scala.Product. Class loader: java.net.URLClassLoader@2eea9be8: java.lang.IllegalStateException: Cannot resolve type description for scala.Product
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:159)
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.toErasure(TypePool.java:6241)
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$Raw$RawAnnotatedType.of(TypePool.java:3412)
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$Raw$RawAnnotatedType$LazyRawAnnotatedTypeList.get(TypePool.java:3511)
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$Raw$RawAnnotatedType$LazyRawAnnotatedTypeList.get(TypePool.java:3456)
at java.base/java.util.AbstractList$Itr.next(AbstractList.java:371)
at kanela.agent.libs.net.bytebuddy.description.type.TypeList$Generic$AbstractBase.accept(TypeList.java:275)
at kanela.agent.libs.net.bytebuddy.dynamic.scaffold.InstrumentedType$Factory$Default$1.represent(InstrumentedType.java:362)
at kanela.agent.libs.net.bytebuddy.ByteBuddy.rebase(ByteBuddy.java:851)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$TypeStrategy$Default$1.builder(AgentBuilder.java:2007)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:10117)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:10494)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10460)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1500(AgentBuilder.java:10223)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10907)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10845)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10413)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(AgentBuilder.java:6865)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:7142)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:6991)
at kanela.agent.libs.net.bytebuddy.agent.bui
But after restart they always disappear, so I assume it's all because subpoject is still compiling. Running it in IntellijIdea