kamon-io/Kamon

The Open Source tool for monitoring applications running on the JVM

Bruno Figueiredo Alves

@brunofigalves

Hi all, I'm using Akka 2.6 and Kamon 2.x and I pretend to gather metrics from Akka and export them to JMX however I don't think it's possible. Could you help me with this or suggest some workarounds?

boriska-ta

@boriska-ta

High all, have a question about cadinality for Kamon tags. Docs say, avoid high cadinality for "metric-related tags". Does "metric-related tag" here means span#tagMetrics - as opposed to Span#tag ? If yes, can I store high cadinality values in Span#tag without generating time series for each value combination ?

Giridhar Pathak

@gpathak

hey folks, i have an older application built on playframework 2.3. does kamon still support that?
or is there an older version of it i can use?

any direction would be helpful.

Tom Milner

@tmilner

Hey, has anyone experienced missing traces when using scala ZIO? I am using Kamon 2.1.4 with OpenTelemetry, and the app is a bit weird, but I basically have a Akka HTTP API, that runs a ZIO, which in turn calls an Akka HTTP Client (I know this is a weird setup). I see the main trace, and spans for things which happens before the ZIO is run but I do not see the traces for the Client call that happened in the ZIO, and it appears the service that was called did not get the trace ID either. I will keep digging, but if anyone has seen something like that or has an idea what it could be I am all ears.

3 replies

Ivan Topolnjak

@ivantopo

@/all hey folks, this is a reminder that we are migrating to Discord for questions and chat related to Kamon. You can join our Discord here: https://discord.gg/5JuYsDJ7au

Have a great week!

Zvi Mints

@ZviMints

i cannot find any metrics exposed via kamon-prometheus

application.conf:

kamon.prometheus {
  include-environment-tags = true
  embedded-server {
    hostname = 0.0.0.0
    port = 9404
  }
}

implementation:

class SinkConnector() extends org.apache.kafka.connect.sink.SinkConnector {
  val underlying: AerospikeSinkConnector = new AerospikeSinkConnector()
  override def start(map: util.Map[String, String]): Unit = {
    Kamon.init()
    Kamon.counter("testing-kamon").withoutTags().increment()
    try {
      underlying.start(map)
    }
    catch {
      case ex: Throwable =>
        println(s"Failure on underlying.start($map)")
        Kamon.counter("underlying-start-connector-failure").withTag("config-file",configFile).withTag("message", ex.getMessage).increment()
        throw ex
    }
    finally {
      Kamon.stopModules()
    }
  }

dependencies:

  "io.kamon" %% "kamon-prometheus" % "2.2.2" exclude("org.slf4j", "slf4j-api"),
  "io.kamon" %% "kamon-core" % "2.1.0" exclude("org.slf4j", "slf4j-api")

I already have JMX Exporter which expose Kafka metrics to 9404, i tried to make Kamon use this port also, when i remove the application.conf and use the default value of port 9095 i cannot port-forward to this port for some reason.

I'm missing something?

Thanks!

Zvi Mints

@ZviMints

Im getting

2021-08-31 12:03:52,224 WARN Failed to attach the instrumentation because the Kamon Bundle is not present on the classpath (kamon.Init) [connector-thread-dashboard-connector-profile]

when i'm using "io.kamon" %% "kamon-prometheus" % "2.2.2" exclude("org.slf4j", "*") - any ideas why?

1 reply

Thomas Jaeckle

@thjaeckle

Hi. We are experiencing the following WARN message:

Failed to record value [-401488] on [span.processing-time,{operation=serialize,error=false}] because the value is outside of the configured range. The recorded value was adjusted to the highest trackable value [3600000000000]. You might need to change your dynamic range configuration for this metric

So the recorded value is negative. What we use is the Kamon SpanBuilder.start(Instant), however the span is later (within sub-milliseconds) finished via Span.finish() (where the underlying Clock is used to determine the nanos of the finish time)
Could it be that this "mixing" can cause negative values being recorded?

Thomas Jaeckle

@thjaeckle

ah, maybe instead of Instant.now() we should use Kamon.clock().instant() which provides better precision/performance?

6 replies

Dana Borinski

@dborinsk

Trying to add kamon metrics to my caffeine cache, can someone explain please the part of KamonStatCounter or give an example? i see this page in the docs Caffeine.newBuilder().recordStats(() -> new KamonStatsCounter("cache_name")).build(); but not sure i understand what is needed to be passed to the recordStats. i see it expects to get a supplier but this example isnt working so i probably miss something.

nikhilaroratgo

@nikhilaroratgo

Can Kamon instrument the play https connections? We have issue in PROD env where the heap memory increased after we deployed with https enabled. Kamon is throwing java.lang.OutOfMemoryError . I suspect that its because Kamon is not able to scrape the https connections. What is your opinion ?

Isaac Povey

@isaacpovey

Anyone know how to initialize Kamon with a play application loaded by guice. I followed the play guide but am getting these errors when trying to load it Caused by: java.lang.VerifyError: Expecting a stackmap frame at branch target 102. The application loader looks like this

class CustomApplicationLoader extends GuiceApplicationLoader {
  override protected def builder(context: Context): GuiceApplicationBuilder =
    super
      .builder(context)
      .eagerlyLoaded()

PrashantN86

@PrashantN86

I am trying to add traceability support to a play 2.8 application with Kamon and Jaeger. I followed [instructions here] (https://kamon.io/docs/latest/reporters/jaeger/) . I am able to see the startup logs for Kanela agent as well as the Jaeger reportes as follows

[info] Running the application with the Kanela agent

 _  __                _        ______
| |/ /               | |       \ \ \ \
| ' / __ _ _ __   ___| | __ _   \ \ \ \
|  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
| . \ (_| | | | |  __/ | (_| |  / / / /
|_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/

==============================
Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.8)

--- (Running the application, auto-reloading is enabled) ---

[info] p.c.s.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9001

(Server started, use Enter to stop and go back to the console...)


2021-09-23 13:29:11,210 [info] [play-dev-mode-akka.actor.default-dispatcher-11] k.i.p.GuiceModule$KamonLoader - Reconfiguring Kamon with Play's Config
2021-09-23 13:29:11,211 [info] [play-dev-mode-akka.actor.default-dispatcher-11] k.i.p.GuiceModule$KamonLoader - play.core.server.AkkaHttpServerProvider
2021-09-23 13:29:11,213 [info] [play-dev-mode-akka.actor.default-dispatcher-11] k.i.p.GuiceModule$KamonLoader - 10 seconds
2021-09-23 13:29:11,573 [info] [play-dev-mode-akka.actor.default-dispatcher-11] k.j.JaegerReporter - Started the Kamon Jaeger reporter

Jaeger is started through a docker container with following command:

docker run -d --name jaeger1   -e COLLECTOR_ZIPKIN_HOST_PORT=:9411   -p 5775:5775/udp   -p 6831:6831/udp   -p 6832:6832/udp   -p 5778:5778   -p 16686:16686   -p 14268:14268   -p 14250:14250   -p 9411:9411   jaegertracing/all-in-one:1.25

None of the traces are visible when I try to access the APIs for my play application. Any configuration I am missing here?

Tommaso Schiavinotto

@Teudimundo

I'm trying to use Kamon-cassandra (v.2.2.2, kanela 1.0.11) . I'm interested in metrics, but the only one I find available at runtime is span_processing_time_seconds. Is there something I need to configure in order to get those listed in the documentation page?

Giridhar Pathak

@gpathak

hey folks... anyone know if the kanela agent works fine with the openj9 jvm?

i find that my apps has heap going up and then OOMs.

Screen Shot 2021-10-13 at 3.33.16 PM.png

the orange line there is the jvm heap.. the heap was set low.. but the container's mem utilization goes up to 1.5Gb and then OOM

does kanela store data off heap?

and how can i debug this/reduce memory usage.

my conf file:

kamon {
  prometheus {
    embedded-server {
      hostname = 0.0.0.0
      port = 9095
    }
    buckets {
      time-buckets = [
        0.25,
        0.5,
        0.75,
        1,
        2.5
      ]
      information-buckets = [
        1024,
        2048,
        4096,
      ]
    }
  }

  instrumentation {

    play {
      server.metrics.enabled = no
      http.server.tracing.enabled = no
      http.client.tracing.enabled = no
    }
  }

  modules.host-metrics.enabled = no
  modules.process-metrics.enabled = no
  modules.status-page.enabled = no
  trace {
    sampler = "never"
    span-metrics = off
    span-metric-tags {
      upstream-service = no
      parent-operation = no
    }
  }
}

kanela.modules {
  akka {
    enabled = no
  }
  akka-remote {
    enabled = no
  }
  akka-remote-sharding {
    enabled = no
  }
}

Libor Kramoliš

@liborkramolis_twitter

Hi. Is it possible to exclude selected HTTP endpoints from producing spans? For example, I would like to hide /ready operation because it is called in regular bases by K8s as readiness probe. I do not need to track such operation. Thanks.

1 reply

Sean Glover

@seglo

good day. i'm having an issue where Kamon is generating strange operation names when there exists an endpoint with more than 1 verb implementation. i'm using guardrail to generate akka-http routes from an openapi spec, and then using Kamon's akka-http server integration. i couldn't find any issues that describe a similar problem, but i thought i would see if anyone here has a pointer.

i.e. i have a GET /foo and POST /foo, an operation name of /foo/foo will be generated for the first defined endpoint from my openapi spec. the second defined endpoint operation name seems unaffected..

1 reply

Sean Glover

@seglo

i created a reproducer PR: kamon-io/Kamon#1063

Zhenhao Li

@Zhen-hao

hi, I'm new to kamon. I inherited a codebase at a client of mine which uses kamon with the datadog-api module. I assume sending metrics data via datadog-api is async and not blocking metrics update, since that's the only sensible way. But it would be nice if people with more experience with kamon can confirm that.

1 reply

Akash Nema

@akash-nema-incontact

Hi Everyone

Can someone help me with Kamon-Jaeger Integration. Jaeger is not reading the Kamon spans I have created in my app. Application details:
Scala version: 2.12.13
Jaeger Version: 2.3.1

play framework version: 2.12
akka: 2.6.10

Akash Nema

@akash-nema-incontact

I'm new to kamon. I have referred https://kamon.io/docs/latest/reporters/jaeger/

I can see the generated span traces in kamon status page but not in jaeger.

Shane

@Shailpat3Shane_twitter

I am getting this error after upgrading kamon-bundle to 2.3.1 from 2.2.1

ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML document. See previously reported errors.
        at ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:65)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:151)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:110)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:53)
        at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
        at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:150)
        at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:84)
        at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
        at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
        at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)

Caused by: org.xml.sax.SAXParseException; systemId: jar:file:/Users/shaileshpatil/workspace/xxxx-jar!/logback.xml; lineNumber: 42; columnNumber: 2; The markup in the document following the root element must be well-formed.
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1243)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324)
        at ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:59)
        ... 18 more

Any help will be appreciated

Aditya Maheshwari

@adityamundra

How to get current value of a counter? For example -

Kamon
          .counter("requests.status.404")
          .withTag("Reason", "The requested resource could not be found.")
          .increment()

After calling increment how to check if the value was incremented?

Zvi Mints

@ZviMints

Hey, I'm facing an issue with the following exception:

2021-11-29 17:44:41.710 31 WARN  MacOperatingSystem:365 - Failed syctl call for process arguments (kern.procargs2), process 42643 may not exist. Error code: 22

Dependencies:

  "io.kamon" %% "kamon-core" % "2.4.1",
  "io.kamon" %% "kamon-bundle" % "2.0.6",
  "io.kamon" %% "kamon-prometheus" % "2.0.1",

I'm trying to disable these errors with the following configurations on application.conf:

kamon.modules {
  host-metrics {
    enabled = no
  }
}

But its not working, any ideas?
cc: @ivantopo (:pray:)

Mayank Srivastava

@mayanksriv

Hi @ivantopo !! I have recently implemented Kamon Instrumentation to my play application and expect to log traceIds/SpanIds/Context-keys, etc with logback. All of it is built into a docker image. But I faced very weird issue, the same image when run in a container would sometimes work properly and sometimes just skip logging traceId/spanId/context-key etc. This was fairly intermittent. On turning debug mode on for Kanela, I saw both logback and play-framework modules had order=2 and competed to be loaded. When logback loaded before play, everything worked fine but if it loaded after play that is when we noticed missing logback entries. The way I have worked around is to explicitly give different orderings to both above modules. But is that expected behaviour to have two modules compete to be loaded.

Ivan Topolnjak

@ivantopo

@/all hey folks, this is a reminder that the official Kamon folks are now giving support on Discord and not checking this room at all. Still some other folks from the community might help here, but if you want to ensure we take a look at your question please drop it on Discord or Github. Thanks!

mofei100

@mofei100

com.mysql.jdbc.PreparedStatement with message Class versions V1_5 or less must use F_NEW frames.. Class loader: jdk.internal.loader.ClassLoaders$AppClassLoader@9e89d68: java.lang.IllegalArgumentException: Class versions V1_5 or less must use F_NEW frames.

Anybody see this problem?

1 reply

Gaël Bréard

@gbrd

Hi, I'm trying to build with graalvm/native-image but it does not work, I can seee tthis is supported here : https://github.com/kamon-io/Kamon/releases/tag/v2.3.0 but I did not found a doc.
Without option --allow-incomplete-classpath, my build fails. And with this option, I have at runtime : " ERROR kamon.Init - Failed to attach the Kanela agent included in the kamon-bundle"

can/should I use allow-incomplete-classpath ?
should I add kamon bundle in my dependencies (and thus classpath) ?

Ivan Topolnjak

@ivantopo

Hey @mofei100 and @gbrd, please come over to the Discord server, we are not looking at this chat very often: https://discord.gg/weHgVmJYNY

pknowles-9

@pknowles-9

Hi. How can we use kamon-testkit with Java ? Are there any examples or documentation ? I'd like to test Kamon metrics. Thank you

Oleksandr Shevchenko

@o-shevchenko

Hey
I have a problem with response B3 headers propagation for Kamon http4s.

    implementation("io.kamon:kamon-core_2.13:2.2.3")
    implementation("io.kamon:kamon-http4s-0.23_2.13:2.2.1")
    implementation("io.kamon:kamon-prometheus_2.13:2.2.3")
    implementation("io.kamon:kamon-jaeger_2.13:2.2.3")

Jaeger export works fine, also I see that B3 headers propagated correctly to my http4s service from a previous one but I don't see B3 headers in the server response. kamon.propagation.http.default.entries.outgoing.span should be b3 by default.

kamon {
  environment {
    service = "service2"
  }

  trace {
    join-remote-parents-with-same-span-id = no
    adaptive-sampler {
      groups {
        metrics {
          operations = ["/metrics"]
          rules {
            sample = never
          }
        }
        healthz {
          operations = ["/healthz"]
          rules {
            sample = never
          }
        }
      }
    }
  }

  prometheus {
    start-embedded-http-server = yes
    embedded-server {
      hostname = "0.0.0.0"
      port = 9095
      metrics-path = "/metrics"
    }
  }

  jaeger {
    host = "jaeger-collector.svc.cluster.local"
    port = 14268
  }
}

I found a couple of issues on Github but looks like it has already been resolved.
Any thoughts on what can be wrong?

1 reply

Zvi Mints

@ZviMints

Hey, i'm trying to integrate with p8s and akka metrics.
https://stackoverflow.com/questions/71941845/is-it-possible-to-use-both-kamon-and-prometheus
Can someone take a look?

Oliver Schrenk

@oschrenk

I don't understand the difference of a Context Tag and a Context Entry

Only tags seem to be able to me merged back to MDC ( see https://kamon.io/docs/latest/instrumentation/logback/#copying-context-to-the-mdc ) and when I look at https://kamon.io/docs/latest/core/context/ tag are not mentioned.

Are they both the same (but historically different)?

ALVARO VILAPLANA GARCIA

@avilaplana

Hi, I am using I have a gauge kamon metric named ws_active_connections that contains 3 tags, tag1 with cardinality 417, tag2 with cardinality 25 and tag3 with cardinality 3.
As you can calculate that metric could have 417 25 3 = 31.275 time series. For that reason I have some concerns:

memory issues due to the amount of time series store internally (I think is a TrieMap)
performance and cardinality due to the amount of calls made to lookup and execute increment/decrement operations
Please, can you give me some advise?

SShivama

@SShivama

Hello Everyone.
I am new Scala and Akka. I am getting the below issue. Can anyone help out on this??

native/libsigar-universal64-macosx.dylib' (fat file, but missing compatible architecture (have 'unknown,x86_64', need 'arm64e'))

1 reply

Deepika H

@deepika5555

Hey All,
I am using kamon-akka-http to fetch metrics around the requests, I am currently not seeing metrics related to status code 500 instead the count is incremented for 200 status code,
for e.g. span_processing_time_seconds_count{component="akka.http.server", error="false", http_method="POST", http_status_code="200", instance="localhost:9095", job="Case-actor-metrics", operation="/case/v1/cases", span_kind="server"}

even though POST /case/v1/cases gave me 500 http status code, The metrics counter is incremented for 200 status code on the same route and request method. Is this a bug in the library or Am I missing anything here?

Zhenhao Li

@Zhen-hao

hi all, I notice that questions here are rarely answered.
If your question is for work and your company has a budget for paid consulting, I invite you to create a pairing request on https://pairtime.com. If I can't help myself, I will do my best to find the right people to help you.

Damian Albrun

@insdami

Hi guys, I'm struggling to configure kamon to work along with lightbend cinnamon propagation. One case is http propagation, cinnamon puts in the http client headers Cinnamon-MDC which is values encoded in base64. What I'd like to do is accept that optional header an implement a custom http header codec so I can parse it and add it to the context without overriding the kamon ones if that makes sense.

I'm looking to rollout the change from cinnamon to kamon service to service without losing telemetry. In the docs custom codecs are mentioned, but it seems outdated since the last version doesn't have such trait.

I've tried to find examples in the repo but I'm not having any luck so far

Where communities thrive

People

Repo info

Activity