kamon-io/Kamon

The Open Source tool for monitoring applications running on the JVM

camomileann

@camomileann

@leon-daniel the weird thing is that the same grafana dashboard works for a standalone Prometheus. Looks like kamon+prometheus is a bit different from prometheus standalone

moriyasror

@moriyasror

Hi, I have a very wired issue, I have service which use kamon with jvm-metrics disabled in the configuration.
I deploy the service on a machine and run it with the kanela agent, now I updated the configuration to enable jvm-metrics and restart the service, but it was not enabled.
Only when I enabled jvm-metrics in my code recompile and redeploy the service, the jvm-metrics are enabled.
But if now I will disable it and restart it won't work.
It's look like somehow kamon ignore the configuration file on the machine and relate only to the configuration as it was on the compilation.

I'm using kamon 2.1.0 with scala
Any one have any idea what can cause that?

Kelvin Mungai

@klvmungai

@camomileann you should have a standalone prometheus configured to collect your metrics from kamon. So you can either

Setup a prometheus server and have it pull metrics from localhost:9096.
Setup a prometheus server and have kamon-prometheus push metrics to a push gateway.

Kamon-prometheus only provides a way to get application metrics to prometheus, it does not run an actual prometheus server.

Daniel Leon

@leon-daniel

Has anyone else noticed that for JVM metrics, the jvm.gc histogram doesn't track the old generation?

I get jvm_gc_seconds_bucket{collector="g1-young"..but not a single metric for g1-old

Daniel Leon

@leon-daniel

I took a look and when a new Notification from JMX arrives, in instrumentation/kamon-system-metrics/src/main/scala/kamon/instrumentation/system/jvm/JvmMetricsCollector.scala:64 the Collector for the old gen can't be found

Daniel Leon

@leon-daniel

and one curious thing is that even though the Metrics is defined to have milliseconds:

val GC = Kamon.histogram (
    name = "jvm.gc",
    description = "Tracks the distribution of GC events duration",
    unit = MeasurementUnit.time.milliseconds
  )

, the resulting metric in prometheus seems to be in seconds jvm_gc_seconds_bucket

Daniel Leon

@leon-daniel

I found the answer to my second question. Apparently all values when exported from prometheus are exported/converted in seconds for Time dimension and bytes for Information dimension no matter what MeasurementUnit is defined for the metric

Daniel Leon

@leon-daniel

@ivantopo , I see no test available for kamon-system-metrics. Do you have any idea why I can't see any old gen GC events?

Ivan Topolnjak

@ivantopo

hey folks :wave:

Sergey Morgunov

@ihostage

:wave: We missed you :disappointed:

Ivan Topolnjak

@ivantopo

lol

Sergey Morgunov

@ihostage

It’s true :joy:

Ivan Topolnjak

@ivantopo

still, I'm seeing people helping each other around here, that's super awesome!

not much need for me around here :joy:

@leon-daniel to be honest, I'm not really sure of how it woks with G1, it might be that all GCs are reported as young or that we are actually missing something specific for G1

regarding prom, metrics must be exposed in base units for Prometheus to scrape, see more info here

in the case of time, the base unit is seconds

Chetan Mehrotra

@chetanmeh

For integrating with New Relic whats the recommended way. I see few repos

https://github.com/kamon-io/kamon-newrelic - This is now archived
https://github.com/kamon-io/Kamon/tree/master/reporters/kamon-newrelic - It does not have README detailing how to use
https://github.com/newrelic/kamon-newrelic-reporter - This is also archived

https://kamon.io/docs/latest/reporters/ - It does not list Kamon New Relic reporter. So not sure if its currently supported

Ivan Topolnjak

@ivantopo

hey @chetanmeh, actually those three repositories ended up in the same place (your option number 2) and that's the latest and greatest

the reporter is not yet listed on the website but that reporter is the right one, it was actually developed by folks from New Relic! If you have any questions just drop them here and maybe mention @jkwatson, he will most likely know the right answers in anything related to the NR reporter!

Chetan Mehrotra

@chetanmeh

Thanks @ivantopo for confirming. Was looking for steps and some help doc. Would try to work out using existing README from one of the archived repo

John Watson

@jkwatson

We should move the old README over from the archived project

John Watson

@jkwatson

Hi @chetanmeh . I'm going to connect you with @abhihub , who can help make sure you can get the support you need.

Chetan Mehrotra

@chetanmeh

Thanks John. I am going through the readme and would try to integrate. Would ping once I make any progress. Thanks for reaching out!

John Watson

@jkwatson

great!

abhihub

@abhihub

@chetanmeh Hi Chetan, let me know if you need anything else

1 reply

davidnaughton

@davidnaughton

Hey Guys, im new to Kamon and looking for some tutorials about setting up kamon with AKKA and sending these metrics to an elasticsearch cluster and visualising the metrics in kibana . thanks

Matt Fulgo

@mfulgo

I'm attempting to get Kamon tracing set up with a DataDog reporter for APM. I've noticed the following warning in the DataDog agent logs:

Error getting trace count: "HTTP header \"X-Datadog-Trace-Count\" not found". Functionality may be limited.

I'm wondering if this is a bug and possibly why I'm not seeing every trace get to the dashboard.

Ivan Topolnjak

@ivantopo

@mfulgo have you checked your sampler configuration?

it might be that traces are just not being sampled

Matt Fulgo

@mfulgo

Ah. I'll dig into that. Having read some documentation about Spans always getting captured, I didn't realize that's not the case for the requests/traces as a whole. Thanks.

Ivan Topolnjak

@ivantopo

yeap.. a sampling decision is taken with the very first request and then carried around to all related requests and services

@davidnaughton setting up Kamon for an Akka application is the same as for any other application and the Get Started steps should be enough to get you going. But, there is no reporter for Elasticsearch :confused:

Matt Fulgo

@mfulgo

It looks like the @Segment annotation went away at some point. I was wondering if there's a way to annotate methods to add spans to a trace? Or do we need to add Kamon.withSpan(...) into methods we want to measure?

Matt Fulgo

@mfulgo

@ivantopo - Is there any way to automatically create Spans for all Scala Futures ? (I imagine that could become quite noisy, but it might be worth it...) https://kamon.io/docs/latest/instrumentation/futures/#creating-spans explains how to do it manually, but that would end up being a lot of code I'd have to add in order to get meaningful traces, given all the places in which we create futures...

Matt Fulgo

@mfulgo

Okay... to do what I want to do - instrument a number of classes without having to annotate every method - I'm looking at creating a custom kanela instrumentation module. However, it seems to not be attaching the OnMethodEnter/Exit advice, and my instrumentation module is greyed out on the Kamon status page. (Looking at the kanela debug logging, it loaded my module) Is there something I have to do to activate the module, besides adding my own section to kanela.modules?

30 replies

Alexis Hernandez

@AlexITC

Hi, I'm trying Kamon but got an example that doesn't compile, is there any equivalent method to the refine one?

val operationLatency = Kamon.timer("operation-latency")
  .refine("operation" -> "login")

Alexis Hernandez

@AlexITC

looks like withTag(name, value) seems the way it works now

Alexis Hernandez

@AlexITC

The docs seem to be outdated, similarly Kamon.gauge("users").set(42) doesn't compile, you need to set tags or specify you don't need them to be able to call set

Alexis Hernandez

@AlexITC

for the people that doesn't use AMP, what do you use?

Alexis Hernandez

@AlexITC

another question, has anyone seen this error? [spans-ingestion] request declined, unknown error it started occurring on my server

Matt Fulgo

@mfulgo

@AlexITC - I'm currently trying out Datadog and am pretty pleased with their offering so far. That said, unless you have specific reasons you can't use it, I'd recommend checking out the Kamon hosted APM service. It's probably going to be more cost effective than a lot of other options and will help support the project.

Alexis Hernandez

@AlexITC

You mean self-hosted? I'm trying the Kamon APM service which seems useful but I have problems with the data retention, while it's ok for most metrics I would like to keep some metrics with longer retention periods, I'm even considering to mount everything by myself

3 replies

Paweł Kiersznowski

@pk044

hey guys, how can I turn off the span metrics? I already tried span-metrics = off but it still reports them

Ivan Topolnjak

@ivantopo

hey @pk044, these other spans that you are seeing metrics for.. are they related to the HTTP client/server instrumentation or just some other spans?

Matt Fulgo

@mfulgo

Okay... to do what I want to do - instrument a number of classes without having to annotate every method - I'm looking at creating a custom kanela instrumentation module. However, it seems to not be attaching the OnMethodEnter/Exit advice, and my instrumentation module is greyed out on the Kamon status page. (Looking at the kanela debug logging, it loaded my module) Is there something I have to do to activate the module, besides adding my own section to kanela.modules?

@ivantopo - Any thoughts on this?

Paweł Kiersznowski

@pk044

mainly Akka, I was thinking if it was possible to filter them out somehow? @ivantopo

40 replies

probably with this: https://kamon.io/docs/latest/core/utilities/

Paweł Kiersznowski

@pk044

hmm.. i just tested it (without filtering the span metrics) on a fresh datadog instance and i don't see the span.processing_time metrics.. pretty weird

Where communities thrive

People

Repo info

Activity