kamon-io/Kamon

The Open Source tool for monitoring applications running on the JVM

Zvi Mints

@ZviMints

With the following configurations, does any one knows why?

kamon {
  datadog {
    hostname = datadog
    port = 8125
    application-name = ${?app.name}
  }
  metric {
    tick-interval = 1 seconds
    track-unmatched-entities = yes
  }
}

Daniel Leon

@leon-daniel

can someone take a look over kamon-io/Kamon#815, please?

@ivantopo , I implemented the solution I suggested to have access to the internal Kafka metrics by implementing the MetricsReporter listener from kafka-client.

PS: is there a repository for the kamon documentation, so I can add there a PR as well with the proposed addition to kamon-kafka ?

3 replies

Tim Spence

@TimWSpence

We’re running apps with kamon in Docker. If we base our apps on openjdk:8-jre-slim then we get the following errors:

java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at kamon.Init.attachInstrumentation(Init.scala:57)
    at kamon.Init.attachInstrumentation$(Init.scala:52)
    at kamon.Kamon$.attachInstrumentation(Kamon.scala:19)
    at kamon.Init.init(Init.scala:43)
    at kamon.Init.init$(Init.scala:42)
    at kamon.Kamon$.init(Kamon.scala:19)
    at com.itv.vindler.Main$.<clinit>(Main.scala:17)
    at com.itv.vindler.Main.main(Main.scala)
Caused by: java.lang.IllegalStateException: No compatible attachment provider is available
    at kamon.lib.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:416)
    at kamon.lib.net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:248)
    at kamon.lib.net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:223)
    at kamon.lib.net.bytebuddy.agent.ByteBuddyAgent.attach(ByteBuddyAgent.java:210)
    at kamon.bundle.Bundle$.$anonfun$attach$3(Bundle.scala:34)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
    at kamon.bundle.Bundle$.withInstrumentationClassLoader(Bundle.scala:88)
    at kamon.bundle.Bundle$.attach(Bundle.scala:34)
    at kamon.bundle.Bundle.attach(Bundle.scala)

If we base it on openjdk:8-jdk-slim instead then it’s fine. Does anyone have any suggestions? I didn’t think that the actual runtime would differ between the jre and jdk, the jre just shouldn’t have javac etc on the path

Ivan Topolnjak

@ivantopo

hey @TimWSpence

Tim Spence

@TimWSpence

Hey @ivantopo :)

Ivan Topolnjak

@ivantopo

having a JDK is a requirement only for attaching the agent in runtime, which is exactly what happens when you have the Kamon Bundle and call Kamon.init()

Tim Spence

@TimWSpence

Thanks. Why is a jdk necessary? And is there a way to do things at compile time instead?

(may be a stupid question)

Ivan Topolnjak

@ivantopo

I don't know the details of why the specific parts that we need for that are not shipped on the JRE, but they are not! Those attachment providers are only availably on the JDK

not a stupid question!

so, if you want to use the JRE instead of a JDK, then add the -javaaagent:/path/to/kanela.jar startup parameter and it should work. If you are using SBT, I recommend using the sbt-javaagent plugin. There is a bit extra info here: https://kamon.io/docs/latest/guides/installation/setting-up-the-agent/

Tim Spence

@TimWSpence

haha thanks. AFAIK we only use it for metrics, rather than any automatic instrumentation. Is there a way we can initialize it that avoids needing the jdk? I prefer not to put that in production if possible

oh :+1:

Ivan Topolnjak

@ivantopo

regarding doing all the instrumentation in compile time, I know there is a lot of work going on to support it in ByteBuddy, which is what we use under the hood for instrumentation

Tim Spence

@TimWSpence

oh nice. Yeah, that would be awesome

Ivan Topolnjak

@ivantopo

I guess the GraalVM trend is bringing us there!

I very much would like to switch everything to build time, but that will take some extra time.. we will see!

Tim Spence

@TimWSpence

Good luck! :)

Tim Spence

@TimWSpence

btw @ivantopo I discovered a bit of the answer as to why the jdk was necessary. We were using kamon-bundlewhich autoloads the kanela agent but this requires the jvm attach API, which is only shipped with the jdk and not the jre. Which is why (as you said) explicitly enable the agent using -javaagent allows you to use the jre instead :)

Declan

@d-g-n

hello all, we've recently updated our stack from using kamon 0.6 to 2.0 (big jump i know), we use the statsd reporter to ship metrics to an instance of statsd which forwards them to graphite. we've verified that the stats are being sent from the box using tcpdump and we've verified that the statsd box is correctly receiving those metrics, they look something like this metric.path=87.0|ms, for some reason ever since we did the update even though the metrics still appear in tcpdump, around 90% of them are getting lost between statsd and graphite, given that the only thing we changed that seems to impact this is the kamon version we're somewhat stumped. does anyone have any ideas? thanks in advance

Declan

@d-g-n

i've tried to grab the udp message that tcpdump is reporting that kamon is sending externally and then i've sent a udp message myself to the same destination with the same content, doing that makes it appear in graphite. i don't understand why the generated packet from kamon-statsd doesn't seem to get picked up

Ivan Topolnjak

@ivantopo

hey @decyg!

do you maybe have the tcpdumps of both messages?

and, do you see anything interesting in the drops column of cat /proc/net/udp?

Declan

@d-g-n

the drops column is "0" for the three entries listed, i do have the dumps but unfortunately i can't share them as-is due to them containing some internal names/info but generally, each packet is 900-1000 bytes, starts with the usual headers and then contains one or more lines of kamon host/jdbc generated stats with our histograms interspersed within. i copied the body of one of the real generated tcpdumps into a file and sent is using cat test.dump > /dev/udp/our.ip.goes.here/8125 and noticed that it was successfully received and processed

thanks by the way, i really appreciate the assistance

Ivan Topolnjak

@ivantopo

never mind, we are happy to help!

these UDP issues are also happening with the Datadog agent and I have been trying to figure out what is the issue

Declan

@d-g-n

for a bit more information, we're running m5.large boxes from aws, the main thing that surprises/confuses me is that sending the packet manually seems to work fine, is there some subtle difference with the packet constructed by kamon as to one just pipes to /dev/udp as above?

Ivan Topolnjak

@ivantopo

one of the hypothesis I had was that UDP packets were being dropped because of how Kamon sends them: all at once. Every minute, when the tick arrives, Kamon sends all the UDP packets in one big wave.

Declan

@d-g-n

i see, is there any way to space that out or to decrease/increase the tick rate?

Ivan Topolnjak

@ivantopo

there is no built-in way to space them, but we could do that! spacing wouldn't change things too much I think, it might even increase the overall traffic

do you see anything that might support that hypothesis above? that UDP packets are indeed being sent, but dropped because there's too many of them at once?

Declan

@d-g-n

curiously i've enabled tcpdump on the other side (our statsd box) and tried to confirm that it's receiving the stats and it seems to be getting the same packets that the source box is sending, just for some reason when it receives a packet that i send manually, it's fine, but a kamon sourced packet doesn't seem to get processed properly, i can't see anything in the logs for statsd that could indicate why the kamon sourced ones are being ignored

Ivan Topolnjak

@ivantopo

and just to be sure, all was fine with 0.6?

Declan

@d-g-n

yeah

Ivan Topolnjak

@ivantopo

back then we were using Akka for sending out those messages, it might have introduced a little bit of jitter in the messages because of jumping across a couple actors before writing to the wire

Declan

@d-g-n

i'm still somewhat convinced that it's the contents of the packet, i tried to spam sending the manual packet that was about 950~ bytes and each instance was received and processed

Ivan Topolnjak

@ivantopo

but if it was something on the packet itself, why would it work when you send the very same package manually?

Declan

@d-g-n

i'm worried i might be missing something in the header though, since it was just the body i sent

Declan

@d-g-n

a note though, in our old kamon 0.6 config we only sent histogram and counter via the subscriptions, but in 2.0 we're sending significantly more by way of the fact that subscriptions doesn't seem to be a concept anymore

could the volume of the stats impact it?

Ivan Topolnjak

@ivantopo

that's exactly the fear, a lot more packets than before, in a very short amount of time

Declan

@d-g-n

do you know why the subscription mechanism was removed? i was looking for the diff but it didn't seem to say why

Ivan Topolnjak

@ivantopo

@decyg what I remember about it is that very few people were using it, and when we moved away from Akka in core, there were no more real subscriptions since we didn't have actors underneath anymore

one thing that keeps coming up is people wanting to send less metrics, we we should find a way to make everything configurable, or to make it easy to decide what metrics to send to one reporter vs another

Where communities thrive

People

Repo info

Activity