kamon-io/Kamon

The Open Source tool for monitoring applications running on the JVM

Ivan Topolnjak

@ivantopo

hey @prongs, that issue with Jaeger.. is it just about the order in which the Spans are shown in the UI or the parent-child relationships are wrong?

Alexey Kiselev

@alexeykiselev

Hello, Kamon devs!
I'm trying to understand how Kamon generates host tag. In local application configuration file I see:

kamon {
  enable = yes
  environment.host = "1"

In base application configuration file:

kamon {
  # Set to "yes", if you want to report metrics
  enable = no

  environment {
    service = "xxx"

So, in InfluxDB for Histogram metrics made by Kamon I see:

host =1
instance=xxx@1
service=xxx

But, if an org.influxdb.Point was written directly using the org.influxdb.InfluxDB driver, the metric also contains tag host that equals to hostname of the machine. By the way, application also reports JVM metrics using kamon-system-metrics. Is it possible that Kamon that is not used to produce some metric adds its host to it?

Arjun Karnwal

@arjunkarnwal

Hello, Kamon devs

I am having some issues about using tapir with akka-http backend together with kamon? I observe problem with resolving operation names in the span metrics, and wonder if there's some workaround ? cc @matwojcik I see you have faced this issue before. Do you find a solution for the same ?

Ivan Topolnjak

@ivantopo

hey @alexeykiselev, what Kamon versions are you using there? some of those names sound like from previous Kamon versions! Regarding the host tag, ideally you will leave kamon.environment.host set to auto and only change it if you really need to. For example, in our own deployments we use an environment variable with the actual name of the host since we are running everything in containers. There are a few settings in the InfluxDB reporter to decide whether you want to set the host tag or not

@arjunkarnwal what exactly is the problem you are seeing?

Arjun Karnwal

@arjunkarnwal

@ivantopo when I use tapir with akka-http backend together with kamon, I observer that span metrics like span_processing_time_seconds_count{operation="http.server.request"} whereas if I dont use Tapir I get span_processing_time_seconds_count{operation="api/v1/mycustomerAPIPath"} . I dont know why but when I use tapir, the operation attribute gets overridden i.e. instead of having the api path, it contain "http.server.request".

6 replies

Alexey Kiselev

@alexeykiselev

hey @alexeykiselev, what Kamon versions are you using there?

It's 2.1.0, but configuration may be from older versions.
We set host to internal node ID to show Kamon histograms in Grafana the same as we show metrics created directly. In latter case we just add tag node with the same ID.
I wonder, is it possible in project where Kamon is used and kamon-system-metrics also used that direct calls to org.influxdb.InfluxDB some how polluted with tags that Kamon set.

2 replies

Red Benabas

@red-benabas

Hi Kamon dev! We're upgrading our SBT native packager project from Kamon 1.X to Kamon 2.X, following the steps in the guide https://kamon.io/docs/latest/guides/migration/from-1.x-to-2.0/. We've added Kanela plugin as well as Kanela agent. When I point project to a zipkin sever running on localhost we can see the traces. However, not on remote zipkin server. Has anyone come across this?
Also, is there a way enable DEBUG level logging in kamon.zipkin?

7 replies

Alexis Hernandez

@AlexITC

Has anyone else integrated kamon on a play server that uses grpc clients generated by scalapb? I'm experienced some weird issues, where some grpc calls get invoked twice with no reason, my belief is that kamon may be the cause (I'm still experimenting)

7 replies

Yaroslav Derman

@yarosman

Hello. Does anyone try to use kamon with zio ?

Ilya

@squadgazzz

Hi! Is it possible to check connection status to metrics db with Kamon?

4 replies

Jakub Kozłowski

@kubukoz

Hi, how could I have Kamon-reported spans include the tags from the context?

10 replies

if that's not possible, I can see I can call .tag on the span directly, but I can't read it that way - or can I?

Khal!l

@redkhalil

Hi kamon aficionados, I would like to get your opinion on this subject matter. I've gotten feedback from a colleague that kamon has issues with the byte code generation, that it could damage your artifact, well in short that it's not reliable. And of course he heard this from colleagues who are not around anymore, so I can't verify this. Anybody has had such an experience before?

5 replies

Daniel van der Ende

@danielvdende

Hi, I'm trying to integrate Kamon with an Akka HTTP app. The docs seem to suggest that it should work out of the box (e.g. add dependency to build.sbt, run Kamon.init()). I can see JVM and other system metrics, but 0 Akka HTTP metrics. Any idea what the problem here could be? Thanks!

34 replies

jogi3778

@jogi3778_twitter

Hi,

we use for our Scala application the Kanela agent 1.0.5. Our application runs with openjdk-14.01 in a docker container.
When the kanela agent is initialized it partially stops.

_  __                _        ______
| |/ /               | |       \ \ \ \
| ' / __ _ _ __   ___| | __ _   \ \ \ \
|  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
| . \ (_| | | | |  __/ | (_| |  / / / /
|_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/

==============================
Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.5)
[main] INFO 2020-06-29 12:45:02  Logger : The Module: Executor Service Capture on Submit Instrumentation is disabled
[main] INFO 2020-06-29 12:45:02  Logger : The Module: Akka Remote Instrumentation is disabled
[main] INFO 2020-06-29 12:45:02  Logger : Loading Akka Instrumentation
[main] INFO 2020-06-29 12:45:02  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.EnvelopeInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.SystemMessageInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.RouterInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.ActorInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.ActorLoggingInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.AskPatternInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.EventStreamInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.ActorRefInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.akka_25.DispatcherInstrumentation
[main] INFO 2020-06-29 12:45:03  Logger :  ==> Loading kamon.instrumentation.akka.instrumentations.akka_26.DispatcherInstrumentation
[main] INFO 2020-06-29 12:45:04  Logger : Loading Executor Service Instrumentation
[main] INFO 2020-06-29 12:45:04  Logger :  ==> Loading kamon.instrumentation.executor.ExecutorTaskInstrumentation

Without docker we don't have this problem. With openjdk-13 it also works with docker. Does anyone know this behaviour?

9 replies

Yaroslav Derman

@yarosman

Hello. Does anyone try to use kamon with zio ?

@ivantopo Have you tried to run kamon with zio, because I don't have context propagation and correct span creation ((

Ivan Topolnjak

@ivantopo

no, didn't try it myself

didn't even try ZIO yet

jmendesky

@jmendesky

Hi, is there any interest to allow Kamon's Context to be serialised in HTTP according to the W3C Correlation Context proposal? https://w3c.github.io/correlation-context/
I could see this as a big plus for interoperability in a mixed-tech architecture. It's not standardised yet, but once finalised, would you consider it/would you be open for a PR or does something speak against it in general?
I could see it implemented similar to Trace Context Codecs so that as a user you could choose how to serialise it.
ATM the only difference in how Kamon serialises context and the W3C proposal is that key-value pairs are separated with ; in Kamon and with , in the proposal (apart from the actual name of the header value - which is configurable already)

5 replies

Franco Albornoz

@dannashirn

Hey guys, so I had a shared library with Kamon 1.x that used the SpanCustomizer which I am trying to replace now with a PreStartHook. The thing is I used to have the customizer with a constructor with two parameters which were used to then customize the spans. The thing is PreStartHook needs to have a parameterless constructor, but I'm not sure how I'd be able to customize the spans more dynamically with it.

6 replies

jmendesky

@jmendesky

Hi all, is there a preferred way to propagate trace contexts and context tags via GRPC? I can see this issue has been around for a while kamon-io/Kamon#616 (assuming this is for akka-grpc) and there is also this repo: https://github.com/nezasa/kamon-akka-grpc. Is there a plan for the future?

4 replies

Ilya

@squadgazzz

Hello, Kamon dev. Here’s my config https://controlc.com/cf392b75. I can’t send anything to influxdb with Kamon because of

ERROR kamon.influxdb.InfluxDBReporter - Metrics POST to InfluxDB failed with status code [404], response body: {"error":"database not found: \"mydb\””}

error. I can’t understand where did it find mydb? I have db0 in configs.

7 replies

Ramakrishna Hande

@rhande-mdsol

Hi, Is the header name X-B3-TraceId renamed to trace-id ? We were using some old version of zipkin but now after some upgrade we have response having header trace-id. We are using Zipkin version 2.12.3. Thanks in advance

3 replies

Ilya

@squadgazzz

Hello, there! Is it possible to add multiple custom fields to histogram? Not the tags. And is there any way to create a custom metric with Kamon without fields p50.0 p70.0 p90.0 p95.0 p99.0 p99.9 service?

2 replies

Khal!l

@redkhalil

Hi, after upgrading to bundle and prometheus 2.1.2 I'm getting the following exception upon startup

Unable to start Kanela Agent. Please remove -javaagent from your startup arguments and contact Kanela support.: java.lang.NoClassDefFoundError: com/datastax/driver/core/Session

6 replies

TjarkoG

@TjarkoG

I don't want to ask like this but since my pull-request is being ignored completely for almost 2 months i don't know how else to contact. Is there something that i forgot to do?

17 replies

Alexis Hernandez

@AlexITC

BTW, the gitter activity on the right hasn't been updated for a year

1 reply

Alexis Hernandez

@AlexITC

Last item is from 2019

Ivan Topolnjak

@ivantopo

ha, didn't see that before

Enzo Bonggio

@enzobonggio

Hi I’m using kanela-agent on a play project and I want to add support for httpok. So I added on my build.sbt -> "io.kamon" %% "kamon-okhttp" % "1.0.3” and the configuration to enable it as it says on reference.conf
The error that I’m facing right now is the following

Caused by: java.lang.ClassNotFoundException: kanela.agent.api.instrumentation.KanelaInstrumentation
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at kanela.agent.util.classloader.ChildFirstURLClassLoader.loadClass(ChildFirstURLClassLoader.java:58)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at kanela.agent.InstrumentationLoader.lambda$loadInstrumentation$b85f757d$1(InstrumentationLoader.java:58)
    at kanela.agent.libs.io.vavr.control.Try.of(Try.java:75)
    at kanela.agent.InstrumentationLoader.loadInstrumentation(InstrumentationLoader.java:56)
    at kanela.agent.InstrumentationLoader.lambda$null$1(InstrumentationLoader.java:46)
    at kanela.agent.libs.io.vavr.collection.List.flatMap(List.java:898)
    at kanela.agent.InstrumentationLoader.lambda$load$4(InstrumentationLoader.java:46)
    at kanela.agent.libs.io.vavr.collection.List.map(List.java:1038)
    at kanela.agent.InstrumentationLoader.load(InstrumentationLoader.java:43)
    at kanela.agent.Kanela.lambda$null$0(Kanela.java:79)
    at kanela.agent.util.classloader.InstrumentationClassPath.use(InstrumentationClassPath.java:84)
    at kanela.agent.Kanela.lambda$start$1(Kanela.java:68)
    at kanela.agent.util.Execution.timed(Execution.java:32)
    at kanela.agent.util.Execution.runWithTimeSpent(Execution.java:41)
    at kanela.agent.Kanela.start(Kanela.java:67)
    at kanela.agent.Kanela.premain(Kanela.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)

Should I add another dependency ? Also I could not find any examples of this so I’m lost

3 replies

Krunoslav Uzelac

@Reeebuuk

Hi guys, I'm having the Unable to start Kanela Agent. Please remove -javaagent from your startup arguments and contact Kanela support issue

javaAgents += "io.kamon" % "kanela-agent" % "1.0.5",

addSbtPlugin("io.kamon" % "sbt-kanela-runner" % "2.0.6")
addSbtPlugin("com.lightbend.sbt" % "sbt-javaagent" % "0.1.5")

any hints? :)

will try 2.1.3

Ivan Topolnjak

@ivantopo

hey @Reeebuuk, indeed, try 2.1.3

Pavel Pliachystsik

@pavel.pliachystsik_gitlab

Hello everybody! I have Play framework application (2.8.1) and subproject with Akka cluster (2.6.3). I'm trying to integrate kamon, added play agent and javaagent to pluguins, enabled it in root project and see some attach errors:
[Attach Listener] ERROR 2020-07-10 09:23:15 Logger : Error => akka.dispatch.Envelope with message Cannot resolve type description for scala.Product. Class loader: java.net.URLClassLoader@2eea9be8: java.lang.IllegalStateException: Cannot resolve type description for scala.Product
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:159)
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.toErasure(TypePool.java:6241)
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$Raw$RawAnnotatedType.of(TypePool.java:3412)
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$Raw$RawAnnotatedType$LazyRawAnnotatedTypeList.get(TypePool.java:3511)
at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$Raw$RawAnnotatedType$LazyRawAnnotatedTypeList.get(TypePool.java:3456)
at java.base/java.util.AbstractList$Itr.next(AbstractList.java:371)
at kanela.agent.libs.net.bytebuddy.description.type.TypeList$Generic$AbstractBase.accept(TypeList.java:275)
at kanela.agent.libs.net.bytebuddy.dynamic.scaffold.InstrumentedType$Factory$Default$1.represent(InstrumentedType.java:362)
at kanela.agent.libs.net.bytebuddy.ByteBuddy.rebase(ByteBuddy.java:851)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$TypeStrategy$Default$1.builder(AgentBuilder.java:2007)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:10117)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:10494)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10460)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1500(AgentBuilder.java:10223)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10907)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10845)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10413)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(AgentBuilder.java:6865)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:7142)
at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:6991)
at kanela.agent.libs.net.bytebuddy.agent.bui

But after restart they always disappear, so I assume it's all because subpoject is still compiling. Running it in IntellijIdea

ang-mic

@ang-mic

Hi everyone, happy Sunday :) I am going through this tutorial that involves Zipkin and Prometheus. When I go to the URL for Prometheus, I can see the text but not the UI. What am I doing wrong? Thanks in advance.

Pavel Pliachystsik

@pavel.pliachystsik_gitlab

@ang-mic prometheus is standalone app, what kamon do, is export it in vew which prometheus can collect. You need to install prometheus to see ui, and grafana to ger great ui for your monitorring tool

ang-mic

@ang-mic

@pavel.pliachystsik_gitlab Thank you for coming back to me. That clears the water, I wll look into it :)

Alex

@forrana

Hi.

  # kamon.context.codecs.string-keys {
  #   request-id = "X-Request-ID"
  # }

Didn't managed to find anywhere - is it possible to setup fallback values for those?

Ivan Topolnjak

@ivantopo

hey

Alex

@forrana

The only way I've managed to do it - is to check the key in Kamon.currentContext and then create a new context with a key(if it's None) which then I use in a wrapper on HTTP call. Seems to me there should be an easier way?

Ivan Topolnjak

@ivantopo

correct me if I'm wrong, but are you asking if there is a way to generate a value for "request-id" if there was none in the current context?

Alex

@forrana

So the idea is I need to propagate a header "X-Request-ID" but if it's not set there should be a default value, yes.

Ivan Topolnjak

@ivantopo

there is no built-in way to go about it

Where communities thrive

People

Repo info

Activity