kamon-io/Kamon

The Open Source tool for monitoring applications running on the JVM

Srepfler Srdan

@schrepfler

where in Zagreb are you located?

the site doesn't say

6 replies

jgsastre

@jgsastre

Hi everyone! I would like to make use of Datadog Metrics Metadata that is possible to set using their HTTP API https://docs.datadoghq.com/api/v1/metrics/#edit-metric-metadata. Indeed it seems like kamon-core has kind of support for that as when you define a metric you have an overload of the constructor for setting its description and units just like you have in that Datadog metadata. But at the end, using the DogStatsD reporter that information seems is not sent to Datadog (indeed seems its not possible to send it through that interface). But looking at the Kamon Reporter for Datadog HTTP API in source code, seems the code is not prepared to handle that description and unit information. That would be great as it will allow to define in one place the information for a Datadog Metric (Description and units). Can anyone confirm this or am I missing something?

1 reply

Matt Dziuban

@mrdziuban

hi all, should I expect the kamon-cats-io module to automatically instrument code using cats.effect.IO or does it require manual instrumentation?

1 reply

Michael Sluyter

@msluyter

Hi,
I was attempting to integrate Kamon with a Lagom application. Following the steps for a play based app, I'm getting this compile error:


 _  __                _        ______
| |/ /               | |       \ \ \ \
| ' / __ _ _ __   ___| | __ _   \ \ \ \
|  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
| . \ (_| | | | |  __/ | (_| |  / / / /
|_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/

==============================
Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.5)

[error] (run-main-0) java.lang.NoClassDefFoundError: java/sql/Date
[error] java.lang.NoClassDefFoundError: java/sql/Date

I'm presuming this is because I'm using openjdk 11. I've read, as a general workaround for the above, that you can add something like --add-modules java.sql,java.xml.bind to your java opts. But I'm guessing I need to add these for the kanela run itself (adding them to the sbt javaOptions doesn't seem to help). Is that possible, and/or anyone know how to do that?

Thanks,
Mike

zongwu

@zongwu233_twitter

hello guys ,I am integrating kamon with Java SpringBoot appliction which has Akka actors , I succeeded in following the instructions of the docs on my MacOS , then I deploy the application to a CentOS7 environment
, all metrics of Akka ,executors were failed. I see the status page, they were disappear. Only system metrics works. I has set the jvm options -javaagent:xxxxx.jar ,and see the Kanela logo in the consloe, the environment is CentOS7, openJDK8 . I try kamon version 2.1.4 and 2.1.9 , Akka version is 2.5.0 . How can I continue troubleshooting ? thx

2 replies

piaobeizu

@piaobeizu

hi, when i started a kanela agent, i sended an request to play framework, i got an error:
Uncaught error from thread [play-dev-mode-akka.actor.default-dispatcher-7]: better.files.package$.FileOps(Ljava/io/File;)Lbetter/files/Implicits$FileOps;, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[play-dev-mode]
java.lang.NoSuchMethodError: better.files.package$.FileOps(Ljava/io/File;)Lbetter/files/Implicits$FileOps;
at kamon.instrumentation.sbt.play.KanelaReloader.$anonfun$reload$3(KanelaReloader.scala:474)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:162)
at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:160)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1429)
at kamon.instrumentation.sbt.play.KanelaReloader.$anonfun$reload$1(KanelaReloader.scala:476)
at kamon.instrumentation.sbt.play.KanelaReloader$$anon$1.run(KanelaReloader.scala:44)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at kamon.instrumentation.sbt.play.KanelaReloader$.kamon$instrumentation$sbt$play$KanelaReloader$$withReloaderContextClassLoader(KanelaReloader.scala:49)
at kamon.instrumentation.sbt.play.KanelaReloader.reload(KanelaReloader.scala:462)
at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:140)
at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:296)
at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:186)
at akka.stream.impl.fusing.MapAsync$$anon$30.onPush(Ops.scala:1261)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:624)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:501)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:599)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:768)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:783)
at akka.actor.Actor.aroundReceive(Actor.scala:533)
at akka.actor.Actor.aroundReceive$(Actor.scala:531)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:690)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:573)
at akka.actor.ActorCell.invoke(ActorCell.scala:543)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:269)
at akka.dispatch.Mailbox.run(Mailbox.scala:230)
at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:678)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:47)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

my play framework version is 2.8.3, kamon jars version are:
"io.kamon" %% "kamon-core" % "2.1.9",
"io.kamon" %% "kamon-bundle" % "2.1.9",
"io.kamon" %% "kamon-logback" % "2.1.9"

7 replies

piaobeizu

@piaobeizu

Zvi Mints

@ZviMints

Hey All!

How can i count how much time did some async function occurred?

With sending the right unit (milliseconds) to datadog ?

JulianPani

@julian-pani

Hi guys,
I followed the basic setup for Play Framework. Everything seems to work except that I'm not seeing any traces at all:

I'm using:

// project/plugins.sbt
addSbtPlugin("io.kamon" % "sbt-kanela-runner-play-2.8" % "2.0.6")

// build.sbt dependencies:
"io.kamon" %% "kamon-bundle" % "2.1.0",
 "io.kamon" %% "kamon-apm-reporter" % "2.1.0"

// application.conf
kamon {
  environment.service = "myService"
  apm.api-key = "xxxxxxxxxxxxxxxxx"
}

Any idea what could be the reason?
I made a few hundred requests to make sure it's not due to sampling.

1 reply

JulianPani

@julian-pani

Some more info on the message above:
In the status page http://localhost:5266/#/ I see:

so it looks to me like traces are being sent. Why can't I see them in Kamon APM?

6 replies

Cesar Alvernaz

@calvernaz

Hi, I just notice I was missing the JVM metrics, even tho I have "kamon-system-metrics:2.1.9" and the kamon.modules { jvm-metrics.enabled = yes } enabled

8 replies

what can I possible be missing?

Wolfgang Bauer

@wbwork

Hey! Has anyone tried running Kamon on arm architecture?
Unfortunately the sigar-loader does not include a .so file for arm64/aarch64 and Kamon(1) fails with a NPE

4 replies

Cédric Gourlay

@kidi

Hey there, I open a discussion https://github.com/kamon-io/Kamon/discussions/924 about the usage of aws xray for traces, if anyone have hints or ideas :D

I am not an expert on the different format of zipkin/jaeger and the xray format

Dmitriy Paramoshkin

@dparamoshkin

Hi, I'm trying to setup monitoring for our akka cluster. One node (UI) is and running in tomcat, another nodes (compute) are normal java applications. I run them both using kanela javaagent.
I see message "Running with Kanela, the Kamon Instrumentation Agent" for both nodes.
Issue is if I run 2 compute nodes they can form a cluster, but if run ui node and compute they can't. Ui node will show

akka.remote.artery.Deserializer:90 - Failed to deserialize message from [unknown] with serializer id [17] and manifest [d]. akka.protobufv3.internal.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).

Akka is 2.6.10, Kamon 2.1.9, Kanela agent 1.0.7
Am I missing something obvious?

7 replies

jatcwang

@jatcwang:matrix.org

[m]

Ooh Croatia. Nice :)

Ben Spencer

@dangerousben

hi, I've added kamon to an http4s app and I'm trying to add trace IDs to my logback logs
it works fine for logging within my service code, but not for http4's own access logs, even if I ensure that I add kamon support after the logging middleware
as I understand it middleware layers in http4s are basically just wrappers around the service so I don't understand how this could fail to work

3 replies

I've verified that the kamon context is preserved across IO chains within the service, even when they cross thread boundaries

slavaschmidt

@slavaschmidt

Hi there! I have a beginner question related to a plain application. I followed the guide and put the Kamon.init() in my main object and the dependency into the build.sbt Unfortunately, when I deploy the docker image into the k8s cluster, I see that the instrumentation is not started:

Instrumentation
Disabled, Reporters 1 Started,  Metrics 49 Metrics

. Could you please give me a hint of what I'm missing?

4 replies

morsch

@morsch

Is it normal for Kanela to take >10s to initialize? Is there any way to speed it up?

5 replies

Yaroslav Derman

@yarosman

Hello. Maybe someone can help me, locally all tests passed, but on the git they are failed

Yaroslav Derman

@yarosman

I told about kamon-io/Kamon#926, and I see that master build was failed too

gourishivani

@gourishivani

I apologize if this has been asked before.
We use third party libraries that use Dropwizard. Our code however uses Kamon. What are the options to expose the metrics collected by the third party libraries that use DropWizard? Are there ways to expose the metrics collected by Dropwizard to Kamon so that the kamon metrics reporting infrastructure could be leveraged?

2 replies

xsolo

@xsolo

Hi, I am having issues with the annotation instrumentation, on the status page it says 'Annotation Instrumentation ' - off my config is

implementation 'io.kamon:kamon-bundle_2.13:2.1.10'
    implementation 'io.kamon:kamon-zipkin_2.13:2.1.10'
    implementation 'io.kamon:kamon-akka_2.13:2.1.10'
    implementation 'io.kamon:kamon-akka-http_2.13:2.1.10'
    implementation 'io.kamon:kamon-annotation_2.13:2.1.10'

and the config is kanela modules.annotation.within = ["^com..*"]

2 replies

xsolo

@xsolo

vijayto

@vijayto

Hi There, am trying to get akka metrics using kamon. I am using following versions for bundle and prometheus -> val kamon_bundle = "2.0.5"
val kamon_prometheus = "2.0.1", am using akk version: 2.5.23. We could get metrics neat and clean when am running with java -jar for our executable.

val kamonPrometheusReporter = PrometheusReporter()
            Kamon.registerModule("prometheus-reporter", kamonPrometheusReporter)
            Kamon.init()

. However the moment am trying to bring up the app in my docker which is using amazon correto JRE, am not able to see akka metrics available at my prometheus server end point. I tried to bring up using kanela agent as well, version 1.0.4 java -javaagent:kanela-agent-1.0.4.jar, any pointers as to what could have gone wrong?

1 reply

Paolo Fabbro

@NewP8

Hi, I'm trying to instrument a microservice application using scala Lagom 1.6 but after configuring the dashboard I can't see actor's metrics.
The configuration that I used is:

val kamonBundle = "io.kamon" %% "kamon-bundle" % "2.1.0"
val kamonApmReporter = "io.kamon" %% "kamon-apm-reporter" % "2.1.0"

lazy val dataobject-impl = (project in file("dataobject-impl"))
.enablePlugins(LagomScala, JavaAgent)
.settings(
...
libraryDependencies ++= Seq(
lagomScaladslPersistenceCassandra,
lagomScaladslKafkaBroker,
...,
kamonBundle,
kamonApmReporter
) ++ kamonDependencies,
javaAgents += "org.aspectj" % "aspectjweaver" % "1.9.2",
javaOptions in Universal += "-Dorg.aspectj.tracing.factory=default"
)

Any advice?

5 replies

hot0cool

@hot0coolTw_twitter

hi! I used the version of Kamon as follows and set it as below.

compile group: 'com.typesafe.akka', name: "akka-actor-typed_2.13", version: "2.6.10"
compile "io.kamon:kamon-bundle_2.13:2.1.4"
compile "io.kamon:kamon-prometheus_2.13:2.1.4"

The operation of the server worked properly.
But what's wrong with not being able to track metrics related to the actor?

9 replies

eyal farago

@eyalfa

hi guys, using akka 2.6.11 + kamon-bundle 2.1.10, I'm seeing repeating class-cast-exceptions related to RoutedActorCell and HasActorMonitor:

An error occurred while trying to apply an advisor: java.lang.ClassCastException: class akka.routing.RoutedActorCell cannot be cast to class kamon.instrumentation.akka.instrumentations.HasActorMonitor (akka.routing.RoutedActorCell and kamon.instrumentation.akka.instrumentations.HasActorMonitor are in unnamed module of loader 'app')
    at kamon.instrumentation.akka.instrumentations.HasActorMonitor$.actorMonitor(ActorInstrumentation.scala:95)
    at kamon.instrumentation.akka.instrumentations.SendMessageAdvice$.onEnter(ActorInstrumentation.scala:136)
    at akka.routing.RoutedActorCell.sendMessage(RoutedActorCell.scala:138)
    at akka.actor.Cell.sendMessage(ActorCell.scala:326)
    at akka.actor.Cell.sendMessage$(ActorCell.scala:325)
    at akka.actor.ActorCell.sendMessage(ActorCell.scala:410)
    at akka.actor.RepointableActorRef.$bang(RepointableActorRef.scala:178)
    at akka.io.SelectionHandler$SelectorBasedManager$$anonfun$workerForCommandHandler$1.applyOrElse(SelectionHandler.scala:118)
    at akka.actor.Actor.aroundReceive(Actor.scala:537)
    at akka.actor.Actor.aroundReceive$(Actor.scala:535)
    at akka.io.SelectionHandler$SelectorBasedManager.aroundReceive(SelectionHandler.scala:101)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
    at akka.actor.ActorCell.invoke(ActorCell.scala:547)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
    at akka.dispatch.Mailbox.run(Mailbox.scala:231)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

according to https://github.com/kamon-io/Kamon/blob/c5396be1437cfb96f912e0e4538afe01b39d930c/instrumentation/kamon-akka/src/common/scala/kamon/instrumentation/akka/instrumentations/ActorInstrumentation.scala#L37 it seems like RoutedActorCell is not instrumented withe the mixin hence it cannot be cast this way

is this a kamon bug?

eyal farago

@eyalfa

seems like the instrumentation doesn't apply to RouterActorCell but since its sendMessage method delegates to super.sendMessage it ends up invoking the adviced code which attempts the cast

override def sendMessage(envelope: Envelope): Unit = {
    if (routerConfig.isManagementMessage(envelope.message))
      super.sendMessage(envelope)
    else
      router.route(envelope.message, envelope.sender)
  }

2 replies

Stela L.

@stela-leon

Hey there, I am using kamon 2.1.9, I get some weird errors every 0.10 seconds : " Failed to get information to use statvfs. path: /root/.cache/gvfs, Error code: 13", I mean it's producing metrics on <>:9095 port, but I would like to fix this, (gave a try to install libfilesys-statvfs-perl << I am using ubuntu 20, didn't work). Anyone seen and solved this?

1 reply

Zvi Mints

@ZviMints

Hey! Im using

 lazy val kamon = Seq(
    "io.kamon" %% "kamon-system-metrics" % "0.6.7",
    "io.kamon" %% "kamon-scala" % "0.6.7",
    "io.kamon" %% "kamon-play-2.6" % "0.6.8",
    "io.kamon" %% "kamon-akka-2.5" % "0.6.8",
    "io.kamon" %% "kamon-datadog" % "0.6.8"
  ).map(_.exclude("org.asynchttpclient", "async-http-client"))

And having the following error on starting application:

java.io.FileNotFoundException: /opt/docker/native/libsigar-amd64-linux.so (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
    at kamon.sigar.SigarProvisioner.provision(SigarProvisioner.java:172)
    at kamon.system.SystemMetricsExtension.<init>(SystemMetricsExtension.scala:52)
    at kamon.system.SystemMetrics$.createExtension(SystemMetricsExtension.scala:34)
    at kamon.system.SystemMetrics$.createExtension(SystemMetricsExtension.scala:32)
    at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1006)
    at akka.actor.ExtensionId.apply(Extension.scala:79)
    at akka.actor.ExtensionId.apply$(Extension.scala:78)
    at kamon.system.SystemMetrics$.apply(SystemMetricsExtension.scala:32)
    at akka.actor.ExtensionId.get(Extension.scala:92)
    at akka.actor.ExtensionId.get$(Extension.scala:92)
    at kamon.system.SystemMetrics$.get(SystemMetricsExtension.scala:32)
    at kamon.ModuleLoaderExtension.$anonfun$new$3(ModuleLoader.scala:43)
    at scala.util.Success.$anonfun$map$1(Try.scala:255)
    at scala.util.Success.map(Try.scala:213)
    at kamon.ModuleLoaderExtension.$anonfun$new$2(ModuleLoader.scala:41)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at kamon.ModuleLoaderExtension.<init>(ModuleLoader.scala:38)
    at kamon.ModuleLoader$.createExtension(ModuleLoader.scala:27)
    at kamon.ModuleLoader$.createExtension(ModuleLoader.scala:25)
    at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1006)
    at kamon.Kamon$Instance._start$lzycompute(Kamon.scala:64)
    at kamon.Kamon$Instance._start(Kamon.scala:58)
    at kamon.Kamon$Instance.start(Kamon.scala:74)
    at kamon.Kamon$.start(Kamon.scala:41)
    at kamon.play.di.GuiceModule$KamonLoader.<init>(GuiceModule.scala:36)
    at kamon.play.di.GuiceModule$KamonLoader$$FastClassByGuice$$d6cb6187.newInstance(<generated>)

How can i disable it?

5 replies

Mark Dufresne

@zwift-mark

Hey folks, anyone know if kamon-datadog supports unix domain sockets ?

Mark Dufresne

@zwift-mark

also, anyone ever run into data-dog agent (same container as app) refusing connection?

2021-02-07 01:35:26.771 ERROR 1 --- [dogSpanReporter] kamon.module.ModuleRegistry              : Reporter [DatadogSpanReporter] failed to process a spans tick.

java.net.ConnectException: Connection refused (Connection refused)

Nikhil Arora

@nikhilaroratgo_gitlab

May I ask if Kamon can also monitor apache http client and okayhttpclient If I use it in Lagom ?

1 reply

hot0cool

@hot0coolTw_twitter

@subrotosanyal hi!. Did you find any way to use Kamon in Tomcat environment?

Channing Walton

@channingwalton

Hi, I've just upgraded kamon from 2.1.10 to 2.1.11 and am getting java.lang.NoClassDefFoundError: akka/http/Version$
The dependencies I have are kamon-bundle and kamon-influxdb.
What could I be missing?

4 replies

hot0cool

@hot0coolTw_twitter

I'd like to link Kanela (1.0.7) to Tomcat (8) , could you give me a detailed guide?

I don't know what to do with the information below.

1.Patching catalina code Bootstrapper,and including Kamon.init there.
2.Include kamon/akka/scala/config libs in a special bootlib dir inside tomcat
3.Configure tomcat catalina.sh CLASSPATH to include bootlib dir
4.Remove kamon/akka/scala/config libs from my webapp WEB-INF/lib dir to prevent colissions
5.Put application.conf file in bootlib dir for kamon configuration

Giridhar Pathak

@gpathak

hey. to get kamon setup, do i need both the kanela agent and the aspectjweaver agent?
i am currently getting random data out of kamon's prometheus reporter. stuff like total actors running == 4881 while the application just spins up 2.

1 reply

this is a simple java application with a main function that spins up an actor system and 2 behaviors. (using akka type 2.6.12)

the metrics looks random.

Where communities thrive

People

Repo info

Activity