kamon-io/Kamon

The Open Source tool for monitoring applications running on the JVM

SimunKaracic

@SimunKaracic

Thank you! I'll answer in the afternoon! :D

przemyslaw wierzbicki

@pshemass

Hey Folks, I configured Akka HTTP tracing and I see all traces in Jaeger. How to exclude some of the traces like /healthcheck?

Diego Parra

@dpsoft

kamon.trace.adaptive-sampler.groups {
health-checks {
operations = [
"healthcheck.get"
]
rules {
sample = never
}
}
}

przemyslaw wierzbicki

@pshemass

Thank you sir!

Rajat Khandelwal

@prongs

Hey guys what do you use for visualizing the metrics?

I'm using grafana and was facing some challenges.

The metric coming from kamon jvm.memory.used is a histogram, not a gauge.

I'm not able to wrap my head around how to plot that in grafana.

Specifically, between these two, which one makes more sense

rate(jvm_memory_used_bytes_sum{job=~"$job", region="heap"}[$interval])/rate(jvm_memory_used_bytes_count{job=~"$job", region="heap"}[$interval])

and

jvm_memory_used_bytes_sum{job=~"$job", region="heap"}/jvm_memory_used_bytes_count{job=~"$job", region="heap"}

Basically Rate(sum)/Rate(count) or simply sum/count

Looking in grafana, the first one goes more up and down and the second one is more of a flat line. I'm not sure which one is best suited here.

SimunKaracic

@SimunKaracic

Hello @prongs , we migrated to github discussions, so please follow up with your question over here -> https://github.com/kamon-io/Kamon/discussions

1 reply

Cesar Alvernaz

@calvernaz

@gipeshka Hi, I had the same problem you described here kamon-io/Kamon#886

After upgrading to kamon 2.1.0 the problem went away

Ales Kozumplik

@akozumpl

Hi Kamoners, what are your thoughts on having a Context key with a mutable value? Is that OK, frowned upon or downright broken?

Use case: store some auditing information so that a Play Action can later verify it's not empty / auditing happened.

Arsene

@Tochemey

Hello folks, I would like to know how I can use Kamon with grpc-java.

Yeah I know someone will suggest that we should be put on gh discussion. It has been there for two days and not a single guidance has been given to it.

João Ferreira

@jtjeferreira

I wrote this and we are using it in prod https://github.com/nezasa/kamon-akka-grpc/releases

(akka-grpc client side uses grpc-java. If you were talking about using grpc-java directly then this is not applicable)

Arsene

@Tochemey

@jtjeferreira thanks for the answer. However I am not using akka grpc. I am using the scalapb-grpc which a scala wrapper around the java grpc.

João Ferreira

@jtjeferreira

ok. what do you want to achieve btw? context propagation? metrics?

Arsene

@Tochemey

@jtjeferreira both. Maybe opentracing will be the right choice.

I have seen some kamon-opentracing project which is kind abandoned .

João Ferreira

@jtjeferreira

Looking at https://scalapb.github.io/docs/grpc/ and io.grpc.stub.AbstractStub I would say you can do the instrumentation yourself in io.grpc.CallCredentials#applyRequestMetadata by passing it to callOptions

Arsene

@Tochemey

@jtjeferreira thanks for the suggestion. much appreciated

Tobias Eriksson

@tobiaseriksson

When I look in Prometheus, I can see JVM stats for GC but surprisingly only for "young"
e.g. jvm_gc_seconds_sum

jvm_gc_seconds_sum{collector="g1-young",component="jvm",country="xxx",customer="xxx",datacenter="dc1",environment="prod",generation="young",instance="xxxx:31882",job="xxx-api-metrics",node="host3"}

How could it be that "Old" generation is missing ?

2 replies

Barnabás Oláh

@stsatlantis

Hello, I wanted to read the (kanela-agent) changelog.md but it looks like it's empty :/ Is it on purpose or the content was accidentally deleted?

Maxim Ivanov

@redbaron

This message was deleted

João Ferreira

@jtjeferreira

https://kamon.io/ is down :(

Darren Bishop

@DarrenBishop

Hi. I’m sure it’s been asked many times but what’s the current approach for surfacing JMX data and exporting with Kamon?

20 replies

Sebastien Vermeille

@sebastienvermeille

Hello guys, I am migrating from kamon 1.x to 2.x and have a small issue regarding SystemMetrics.startCollecting(); which no longer exists. Do you know what is the replacement way of doing that ? Kamon.init() ? Thank you in advance for your precious help

1 reply

Kamon.loadModules()

ok top

Peter Nerg

@pnerg

Getting some really weird results out of the akka metrics. So I decided to do a really simple Ping/Pong app with two static actors and one that gets created and killed every 5 seconds.
The metrics for the active actor count just keeps increasing which makes no sense
akka_system_active_actors_count{system="area51-kamon-akka"} 2865.0
Same goes go for queued messages which keeps increasing but as I only send a Ping and a Pong as response it is impossible to have a queue/inbox growing
The PingerPonger code example
I can't see what I'm doing wrong...pointers would be appreciated

Using latest version of Kamon 2.x, Java 11 OpenJDK and Scala 2.13

2 replies

Oliver

@tupol

Hello everybody, a newbie here in need of help.
I am trying out Kamon and so far I had success instrumenting metrics with Prometheus.
Now I am trying to get to instrumenting traces with Zipkin and I am running into some troubles.
I am going back to basics, following the instructions from Elementary Akka Setup.
I can not get any traces in Zipkin nor I see any logged errors.
Here are a few points about my environment:

scala: 2.13.1
akka: 2.6.8
kamon-bundle: 2.1.6
kamon-zipkin: 2.1.6
Zipkin image: docker run -d -p 9411:9411 openzipkin/zipkin
-javaagent:/..full..path../kanela-agent-1.0.6.jar
I would appreciate any good pointers or redirections.

4 replies

David Pennell

@dpennell

test

Francisco H

@fherrera-automox_gitlab

Hi, I'm new to Kamon and got stuck trying to use it first time. I downloaded the Akka-Quickstart example (https://developer.lightbend.com/guides/akka-quickstart-java/) and then added Kamon following the instructions here - https://kamon.io/get-started/ ... when trying to run both command line and on IntelliJ I am getting this error:

3 replies

Task :run FAILED
Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.Seq$.empty()Lscala/collection/GenTraversable;
at kamon.Configuration.$init$(Configuration.scala:28)
at kamon.Kamon$.<init>(Kamon.scala:19)
at kamon.Kamon$.<clinit>(Kamon.scala)
at kamon.Kamon.init(Kamon.scala)
at com.example.AkkaQuickstart.main(AkkaQuickstart.java:9)
FAILURE: Build failed with an exception.

I would appreciate some help, not sure what else to look at.

Srepfler Srdan

@schrepfler

where in Zagreb are you located?

the site doesn't say

6 replies

jgsastre

@jgsastre

Hi everyone! I would like to make use of Datadog Metrics Metadata that is possible to set using their HTTP API https://docs.datadoghq.com/api/v1/metrics/#edit-metric-metadata. Indeed it seems like kamon-core has kind of support for that as when you define a metric you have an overload of the constructor for setting its description and units just like you have in that Datadog metadata. But at the end, using the DogStatsD reporter that information seems is not sent to Datadog (indeed seems its not possible to send it through that interface). But looking at the Kamon Reporter for Datadog HTTP API in source code, seems the code is not prepared to handle that description and unit information. That would be great as it will allow to define in one place the information for a Datadog Metric (Description and units). Can anyone confirm this or am I missing something?

1 reply

Matt Dziuban

@mrdziuban

hi all, should I expect the kamon-cats-io module to automatically instrument code using cats.effect.IO or does it require manual instrumentation?

1 reply

Michael Sluyter

@msluyter

Hi,
I was attempting to integrate Kamon with a Lagom application. Following the steps for a play based app, I'm getting this compile error:


 _  __                _        ______
| |/ /               | |       \ \ \ \
| ' / __ _ _ __   ___| | __ _   \ \ \ \
|  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
| . \ (_| | | | |  __/ | (_| |  / / / /
|_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/

==============================
Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.5)

[error] (run-main-0) java.lang.NoClassDefFoundError: java/sql/Date
[error] java.lang.NoClassDefFoundError: java/sql/Date

I'm presuming this is because I'm using openjdk 11. I've read, as a general workaround for the above, that you can add something like --add-modules java.sql,java.xml.bind to your java opts. But I'm guessing I need to add these for the kanela run itself (adding them to the sbt javaOptions doesn't seem to help). Is that possible, and/or anyone know how to do that?

Thanks,
Mike

zongwu

@zongwu233_twitter

hello guys ,I am integrating kamon with Java SpringBoot appliction which has Akka actors , I succeeded in following the instructions of the docs on my MacOS , then I deploy the application to a CentOS7 environment
, all metrics of Akka ,executors were failed. I see the status page, they were disappear. Only system metrics works. I has set the jvm options -javaagent:xxxxx.jar ,and see the Kanela logo in the consloe, the environment is CentOS7, openJDK8 . I try kamon version 2.1.4 and 2.1.9 , Akka version is 2.5.0 . How can I continue troubleshooting ? thx

2 replies

piaobeizu

@piaobeizu

hi, when i started a kanela agent, i sended an request to play framework, i got an error:
Uncaught error from thread [play-dev-mode-akka.actor.default-dispatcher-7]: better.files.package$.FileOps(Ljava/io/File;)Lbetter/files/Implicits$FileOps;, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[play-dev-mode]
java.lang.NoSuchMethodError: better.files.package$.FileOps(Ljava/io/File;)Lbetter/files/Implicits$FileOps;
at kamon.instrumentation.sbt.play.KanelaReloader.$anonfun$reload$3(KanelaReloader.scala:474)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:162)
at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:160)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1429)
at kamon.instrumentation.sbt.play.KanelaReloader.$anonfun$reload$1(KanelaReloader.scala:476)
at kamon.instrumentation.sbt.play.KanelaReloader$$anon$1.run(KanelaReloader.scala:44)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at kamon.instrumentation.sbt.play.KanelaReloader$.kamon$instrumentation$sbt$play$KanelaReloader$$withReloaderContextClassLoader(KanelaReloader.scala:49)
at kamon.instrumentation.sbt.play.KanelaReloader.reload(KanelaReloader.scala:462)
at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:140)
at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:296)
at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:186)
at akka.stream.impl.fusing.MapAsync$$anon$30.onPush(Ops.scala:1261)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:624)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:501)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:599)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:768)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:783)
at akka.actor.Actor.aroundReceive(Actor.scala:533)
at akka.actor.Actor.aroundReceive$(Actor.scala:531)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:690)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:573)
at akka.actor.ActorCell.invoke(ActorCell.scala:543)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:269)
at akka.dispatch.Mailbox.run(Mailbox.scala:230)
at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:678)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:47)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

my play framework version is 2.8.3, kamon jars version are:
"io.kamon" %% "kamon-core" % "2.1.9",
"io.kamon" %% "kamon-bundle" % "2.1.9",
"io.kamon" %% "kamon-logback" % "2.1.9"

7 replies

piaobeizu

@piaobeizu

Zvi Mints

@ZviMints

Hey All!

How can i count how much time did some async function occurred?

With sending the right unit (milliseconds) to datadog ?

JulianPani

@julian-pani

Hi guys,
I followed the basic setup for Play Framework. Everything seems to work except that I'm not seeing any traces at all:

I'm using:

// project/plugins.sbt
addSbtPlugin("io.kamon" % "sbt-kanela-runner-play-2.8" % "2.0.6")

// build.sbt dependencies:
"io.kamon" %% "kamon-bundle" % "2.1.0",
 "io.kamon" %% "kamon-apm-reporter" % "2.1.0"

// application.conf
kamon {
  environment.service = "myService"
  apm.api-key = "xxxxxxxxxxxxxxxxx"
}

Any idea what could be the reason?
I made a few hundred requests to make sure it's not due to sampling.

1 reply

JulianPani

@julian-pani

Some more info on the message above:
In the status page http://localhost:5266/#/ I see:

so it looks to me like traces are being sent. Why can't I see them in Kamon APM?

6 replies

Cesar Alvernaz

@calvernaz

Hi, I just notice I was missing the JVM metrics, even tho I have "kamon-system-metrics:2.1.9" and the kamon.modules { jvm-metrics.enabled = yes } enabled

8 replies

what can I possible be missing?

Where communities thrive

People

Repo info

Activity