Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
rpankajakshan
@rpankajakshan

Also tried with 

Seq(
      "io.kamon" %% "kamon-bundle" % "2.0.1",
      "io.kamon" %% "kamon-prometheus" % "2.0.0"
    )

went to http://0.0.0.0:5266
the page shows

INSTRUMENTATION Active REPORTERS 1 Started METRICS 79 Metrics

Jameel Al-Aziz
@jalaziz

Reposting for visibility.

I'm seeing something weird. When I run unti tests with an object that's mocked (but the implemenation uses a static logger), I'm getting the following exception:

Caused by: sbt.ForkMain$ForkError: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at ch.qos.logback.core.spi.AppenderAttachableImpl.<clinit>(AppenderAttachableImpl.java:112)
    ... 73 more
Caused by: sbt.ForkMain$ForkError: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at kanela.agent.libs.net.bytebuddy.dynamic.Nexus.initialize(Nexus.java:137)
    ... 77 more
Caused by: sbt.ForkMain$ForkError: java.lang.IllegalArgumentException: Can not set static kamon.instrumentation.logback.AppendLoopMethodInterceptor$ field ch.qos.logback.core.spi.AppenderAttachableImpl.delegate$u0i5811 to kamon.instrumentation.logback.AppendLoopMethodInterceptor$
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
    at sun.reflect.UnsafeQualifiedStaticObjectFieldAccessorImpl.set(UnsafeQualifiedStaticObjectFieldAccessorImpl.java:81)
    at java.lang.reflect.Field.set(Field.java:764)
    at kanela.agent.libs.net.bytebuddy.implementation.LoadedTypeInitializer$ForStaticField.onLoad(LoadedTypeInitializer.java:124)
    at kanela.agent.libs.net.bytebuddy.implementation.LoadedTypeInitializer$Compound.onLoad(LoadedTypeInitializer.java:187)
    at kanela.agent.libs.net.bytebuddy.agent.builder.AgentBuilder$InitializationStrategy$SelfInjection$Dispatcher$InjectingInitializer.onLoad(AgentBuilder.java:3146)
    ... 82 more

Could this be a bug in kamon-logback? Disabling Kanela worked (we don't call Kamon.init for tests anyway).

karthiravich
@karthiravich

@ihostage The pom.xml has the fabric8 plugin to create docker image . I had purposely created a Dockerfile with the pom.xml contents with little additions for Kanela.jar .

I am giving maven goals - clean package and manually building the docker image for now. Below is the source code of the application for which I am trying to setup Kamon metrics collection

https://github.com/lightbend/akka-java-cluster-openshift.git

@ivantopo I am building the jar using maven - mvn clean package and then running it as below
java -jar <application.jar> . This way I get the Instrumentation to be active in Kamon Status (localhost:5266)

I manually build the docker image using docker build with the same jar and deploy it to Kubernetes cluster . Now when I check the Kamon status in browser, the Instrumentation shows to be Disabled and only JVM and host metrics are showing up - Akka cluster related metrics are not coming up.

Sergey Morgunov
@ihostage

@karthiravich Yesterday you share java.lang.NoSuchMethodError and my question be

Where you configure the classpath for run your akka-cluster-openshift.jar?

And this question is actual :smile: Because I do not see -cp param on your command. Did you set classpath in MANIFEST? What exception did you get today?

Sergey Morgunov
@ihostage

It’s my configuration.
pom.xml

      <plugin>
        <groupId>io.fabric8</groupId>
        <artifactId>docker-maven-plugin</artifactId>
        <version>0.30.0</version>
        <dependencies>
          <dependency>
            <groupId>io.fabric8</groupId>
            <artifactId>run-java-sh</artifactId>
            <version>1.3.4</version>
          </dependency>
        </dependencies>
        <executions>
          <execution>
            <id>push</id>
            <goals>
              <goal>build</goal>
              <goal>push</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <skip>true</skip>
          <images>
            <image>
              <name>foo-bar/service:%l</name>
              <registry>my-registry:8888</registry>
              <build>
                <from>openjdk:8u201-jre-alpine</from>
                <tags>
                  <tag>${env.DOCKER_IMAGE_VERSION}</tag>
                </tags>
                <assembly>
                  <name>opt</name>
                  <descriptor>${maven.multiModuleProjectDirectory}/assembly.xml</descriptor>
                </assembly>
                <optimise>true</optimise>
                <runCmds>
                  <runCmd>apk add udev</runCmd>
                </runCmds>
                <env>
                  <JAVA_MAIN_CLASS>play.core.server.ProdServerStart</JAVA_MAIN_CLASS>
                  <JAVA_CLASSPATH>/opt/lib/*</JAVA_CLASSPATH>
                </env>
                <cmd>/opt/bin/run-java.sh</cmd>
              </build>
            </image>
          </images>
        </configuration>
      </plugin>

assembly.xml

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
  <includeBaseDirectory>false</includeBaseDirectory>
  <dependencySets>
    <dependencySet>
      <useProjectArtifact>true</useProjectArtifact>
      <scope>runtime</scope>
      <fileMode>0644</fileMode>
      <outputDirectory>lib</outputDirectory>
      <excludes>
        <exclude>io.kamon:kanela-agent</exclude>
      </excludes>
    </dependencySet>
    <dependencySet>
      <includes>
        <include>io.kamon:kanela-agent</include>
      </includes>
      <outputDirectory>bin</outputDirectory>
      <outputFileNameMapping>${artifact.artifactId}.${artifact.extension}</outputFileNameMapping>
    </dependencySet>
  </dependencySets>
  <fileSets>
    <fileSet>
      <includes>
        <include>run-java.sh</include>
      </includes>
      <directory>${project.build.directory}/docker-extra/run-java</directory>
      <outputDirectory>bin</outputDirectory>
      <fileMode>0755</fileMode>
    </fileSet>
  </fileSets>
</assembly>

Environment variables for pod (used by run-java.sh):

JAVA_OPTIONS: >-
    -XshowSettings:vm
    -XX:+UnlockExperimentalVMOptions
    -XX:+UseCGroupMemoryLimitForHeap
    -XX:+PrintGCDetails
    -XX:+PrintGCDateStamps
    -Xloggc:/var/log/gc.log
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=10
    -XX:GCLogFileSize=32M
    -javaagent:/opt/bin/kanela-agent.jar
Ivan Topolnjak
@ivantopo
@jalaziz let's track that issue here: kamon-io/kamon-logback#24, I also added a possible workaround (disabling the Logback instrumentation in tests.. more like a hackaround :sweat_smile:)
Jameel Al-Aziz
@jalaziz
Thanks! Didn't file a ticket since I wasn't sure if I was doing something wrong. Good to know it may be an actual bug!
Ivan Topolnjak
@ivantopo
it would be awesome if there was a way to reproduce the issue
is there any quick way to reproduce it?
rpankajakshan
@rpankajakshan
@ivantopo any hint on the kamon-prometheus issue?
Ivan Topolnjak
@ivantopo
@rpankajakshan let me take a quick look at that
Jameel Al-Aziz
@jalaziz

is there any quick way to reproduce it?

Will try to come up with a minimal test to reproduce.

David Padbury
@davidpadbury
Hello 👋 - I seem to be having some difficultly using kanela-agent. I’m going off of what the kamon status page is telling me (all instrumentation items are gray). I initially tried using attachInstrumentation but am now just specifying -javaagent with no luck. Any tips for how to debug what I’m probably doing wrong?
Sergey Morgunov
@ihostage
@davidpadbury As a first step, add next property to your configuration:
kanela {
  debug-mode = true
  log-level = "DEBUG"
}
David Padbury
@davidpadbury
Perfect. Cheers @ihostage.
Sergey Morgunov
@ihostage
Does your log have Kanela Label on start JVM?

 _  __                _        ______
| |/ /               | |       \ \ \ \
| ' / __ _ _ __   ___| | __ _   \ \ \ \
|  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
| . \ (_| | | | |  __/ | (_| |  / / / /
|_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/

==============================
Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.1)
David Padbury
@davidpadbury
It certainly was until I dropped in the config to suppress it :). Should verify i’m still seeing that with the -javaagent approach.
Sergey Morgunov
@ihostage
After enabling debug log, on start JVM you should be seeing many lines like this :
[thread_name] INFO 2019-08-20 18:23:37  Logger : Transformed => class FooBarClass and loaded from sun.misc.Launcher$AppClassLoader
David Padbury
@davidpadbury

@ihostage just had a chance to look now. was able to enable the kanela debug log. Do see a bunch of these messages:

[main] INFO 2019-08-21 15:43:01  Logger : The Module: Executor Service Capture on Submit Instrumentation is disabled 
[main] INFO 2019-08-21 15:43:01  Logger : Loading Logback Instrumentation  
[main] INFO 2019-08-21 15:43:01  Logger :  ==> Loading kamon.instrumentation.logback.LogbackInstrumentation  
[main] INFO 2019-08-21 15:43:02  Logger : Loading Executor Service Instrumentation  
[main] INFO 2019-08-21 15:43:02  Logger :  ==> Loading kamon.instrumentation.executor.ExecutorTaskInstrumentation  
[main] INFO 2019-08-21 15:43:02  Logger : Loading Scala Future Intrumentation  
[main] INFO 2019-08-21 15:43:02  Logger :  ==> Loading kamon.instrumentation.futures.scala.FutureChainingInstrumentation

However no Transformed messages like you referenced.
Then see a bunch of these being printed:

[kamon-agent-strategy-cache-listener-2] DEBUG 2019-08-21 15:44:05  Logger : Expiring key: sun.reflect.DelegatingClassLoader@3228d990with valuekanela.agent.libs.net.bytebuddy.pool.TypePool$CacheProvider$Simple@68d3fa6a
Ivan Topolnjak
@ivantopo
in the case of the Future instrumentation, for example, you should see a message about transforming scala.util.Success/Failure
did you see any of those?
David Padbury
@davidpadbury
No transforming messages at all.
Ivan Topolnjak
@ivantopo
and, I guess this is a stupid question but I need to ask: are you using Try and Futures in your codebase?
I'm wondering if there is a remote possibility that those classes are just not used
or not used "yet"
David Padbury
@davidpadbury
A few :smile:
but let me double check I’m not hitting the “yet” case.
David Padbury
@davidpadbury

Yeah, been through a few ways of running this. Never seeing any Transforming messages. The startup log ends with this:

[main] INFO 2019-08-21 16:40:45  Logger : Reinstrumenter activated.
[main] DEBUG 2019-08-21 16:40:45  Logger : Reinstrumenter is listening for Reinstrumentation Events.
[main] INFO 2019-08-21 16:40:46  Logger : Class Replacer activated.
[main] INFO 2019-08-21 16:40:46  Logger : Startup completed in 3047 ms

Then nothing until those Expiring key messages.

Samuel
@garraspin
Hi! can anyone point me to the config where i can change prometheus endpoint to /metrics in kamon 2.0 ? thanks!
Sergey Morgunov
@ihostage
@garraspin Prometheus has embedded server that listen custom port (9095 by default) and return data by any endpoint, what you want :smile:
All, what you can configure for Prometheus Reporter you can see on reference.conf
Marcelo Carmona
@marcelocarmona
Hi everyone :), I can't see this metric "akka_actor_errors_total" in my Prometheus endpoint
# TYPE akka_system_dead_letters_total counter
akka_system_dead_letters_total{system="default"} 0.0
akka_system_dead_letters_total{system="service-api"} 0.0
# TYPE akka_system_unhandled_messages_total counter
akka_system_unhandled_messages_total{system="default"} 0.0
akka_system_unhandled_messages_total{system="service-api"} 0.0
# TYPE akka_system_processed_messages_total counter
akka_system_processed_messages_total{system="service-api",tracked="false"} 22.0
akka_system_processed_messages_total{system="service-api",tracked="true"} 0.0
akka_system_processed_messages_total{system="default",tracked="true"} 0.0
akka_system_processed_messages_total{system="default",tracked="false"} 40.0
# TYPE akka_http_server_open_connections histogram
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="10.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="30.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="100.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="300.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="1000.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="3000.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="10000.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="30000.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="100000.0"} 1305.0
akka_http_server_open_connections_bucket{interface="0.0.0.0",port="9001",le="+Inf"} 1305.0
akka_http_server_open_connections_count{interface="0.0.0.0",port="9001"} 1305.0
akka_http_server_open_connections_sum{interface="0.0.0.0",port="9001"} 0.0
# TYPE akka_http_server_active_requests histogram
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="10.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="30.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="100.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="300.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="1000.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="3000.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="10000.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="30000.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="100000.0"} 1305.0
akka_http_server_active_requests_bucket{interface="0.0.0.0",port="9001",le="+Inf"} 1305.0
akka_http_server_active_requests_count{interface="0.0.0.0",port="9001"} 1305.0
akka_http_server_active_requests_sum{interface="0.0.0.0",port="9001"} 0.0
# TYPE akka_system_active_actors histogram
akka_system_active_actors_bucket{system="default",le="10.0"} 1.0
akka_system_active_actors_bucket{system="default",le="30.0"} 1.0
akka_system_active_actors_bucket{system="default",le="100.0"} 1335.0
akka_system_active_actors_bucket{system="default",le="300.0"} 1335.0
akka_system_active_actors_bucket{system="default",le="1000.0"} 1335.0
akka_system_active_actors_bucket{system="default",le="3000.0"} 1335.0
akka_system_active_actors_bucket{system="default",le="10000.0"} 1335.0
akka_system_active_actors_bucket{system="default",le="30000.0"} 1335.0
akka_system_active_actors_bucket{system="default",le="100000.0"} 1335.0
akka_system_active_actors_bucket{system="default",le="+Inf"} 1335.0
akka_system_active_actors_count{system="default"} 1335.0
akka_system_active_actors_sum{system="default"} 66691.0
akka_system_active_actors_bucket{system="service-api",le="10.0"} 37.0
akka_system_active_actors_bucket{system="service-api",le="30.0"} 1383.0
akka_system_active_actors_bucket{system="service-api",le="100.0"
does anyone know if it is deprecated or do I need some specific config to add it?
Marcelo Carmona
@marcelocarmona
bay the way, this is the output of my Prometheus endpoint after starting the app
Sergey Morgunov
@ihostage
@marcelocarmona Please check your Kamon status page http://localhost:5266/.
Maybe metric enabled but you just haven't an instance of this metric :smile: Your actors work without errors :smile:
Marcelo Carmona
@marcelocarmona
Thanks, Sergey I'm using Kamon 1 I think I don't have a status page in this version, but yes I was thinking exactly that, I'm trying to generate an error now :+1:
rpankajakshan
@rpankajakshan
@ivantopo if 2.0.x is not quite ready. Will it be possible to cross build 1.x.x for scala 2.13? I am in the process of upgrading my app to scala 2.13 and would help a lot.
Ivan Topolnjak
@ivantopo
@davidpadbury I was thinking about something.. are you actually having issues with Context propagation across Futures? cause, it could be that there is a bug in how we detect whether the module is being used or not instead of in the module actually working!
@rpankajakshan oh it is ready! I think there might be just a small issue on how the image is being built for Kubernetes. We deploy our apps with Docker as well and there are no issues with it
Samuel
@garraspin
Thanks for the info @ihostage ! But how can i have my service providing the /metrics endpoint (instead of the embedded server)
Before I could do it by doing new PrometheusReporter().scrapeData in one endpoint of my service
of course i needed to add the reporter to Kamon with Kamon.addReporter(prometheusReporter)
but Kamon2 hasn't got the addReporter method
rpankajakshan
@rpankajakshan
@garraspin kamon2 loads reporters automatically - https://kamon.io/docs/latest/guides/migration/from-1.x-to-2.0/

@ivantopo if 2.0.x is not quite ready. Will it be possible to cross build 1.x.x for scala 2.13? I am in the process of upgrading my app to scala 2.13 and would help a lot.

Ivan I am referring to the kamon prometheus problems I am having, that I reported earlier in the thread. Did you get a chance to look at this?

Dominic Scheirlinck
@dominics

Using Kamon 2.0, I have some specs2 tests in Play 2.7 that pull up the app (with GuiceApplicationBuilder) and tear it down again (functional tests basically). With these, I see that the ModuleRegistry's stop gets called from the application lifecycle (from the Kamon GuiceModule), and it dutifully shuts down the _metricsTickerExecutor and _spansTickerExecutor.

But I can't see where they ever get restarted - this leads to, on the next start of the app:

2019-08-25T22:34:18.235+0000 [error] kamon.Configuration specs2-7 - Exception occurred while trying to run a OnReconfigureHook
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@695c332d rejected from java.util.concurrent.ScheduledThreadPoolExecutor@1b70d3a8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
    at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
    at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:573)
    at kamon.module.ModuleRegistry.scheduleMetricsTicker(ModuleRegistry.scala:191)
    at kamon.module.ModuleRegistry.reconfigure(ModuleRegistry.scala:121)

(Which makes sense, because after terminating those scheduled executors, it'll never again call Executors.newScheduledThreadPool)

So, the ModuleRegistry is can only be used once at the moment? The assumption seems to be that a ModuleRegistry will not be used again once stop is called on it - but ModuleLoading doesn't seem to recreate its _moduleRegistry instance either - so, should I be looking into the Init stuff? Is there a way to cleanly shut Kamon down within a process such that it can be restarted later? For now, the simple workaround is "don't run Kamon anywhere a Play application is restarted in the same process", but I was wondering if there was a better way. (Didn't need to worry about disabling a module with Kamon 1.x)

Dominic Scheirlinck
@dominics
There are two other things I'm running into with the JDBC and logback instrumentation - slick support and a failure to convert an IAccessEvent into HasContext respectively, but I'll wait to detail them after I hear about that ^
Dominic Scheirlinck
@dominics
(logback one just needed adding kanela.modules.logback.within += "org.databrary.logback..*" because I'm using https://github.com/cardamo/play-logback-access - fixed!)