Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Jonas Kunz
    @JonasKunz
    You have to options to reduce the number of traces:
    cattox
    @cattox
    Hi all, I'm trying to do something that I believe is quite simple, but I'm struggling to find the correct configuration for it. I've spent some time reading the documentation and searching the web with little success. I have my scenario working as I wanted except for a little detail. I want to consider only the spans that take more than 10ms to complete. Can you point me out the correct way to achieve it? Thanks in advance.
    cattox
    @cattox
    I know that the previous message talks about keeping only slow traces, but what I really want is to keep all traces only with slow spans.
    Jonas Kunz
    @JonasKunz

    Unfortunately, here the problem is the same: With OpenCensus / OpenTelemetry / OpenTracing, a span needs to know whether or not it shall be sampled at thepoint in time when it is started.

    Why is this necessary?
    When a span is started, it makes itself the parent of all following spans. Therefore it already needs to know at this point, if it should make itself the parent of following spans or not.

    RRadziejewski
    @RRadziejewski
    How to add some tracing config which can be assigned to every project so i can basically test if traces are working, Im on this from 2 days, please help
    @JonasKunz Can you please help ?
    Jonas Kunz
    @JonasKunz
    As stated in git, by using the configuration provided with the demo you should be able to see traces.
    Alternatively yo ucan wait until we release version 1.3 this week, which will include the tracing configuration in it's core. This means then no additional configuration except for the jaeger exporter is required to see HTTP traces.
    Jonas Kunz
    @JonasKunz
    *in github
    RRadziejewski
    @RRadziejewski
    When will be the release
    ?
    unknownuu
    @unknownuu
    How do we get method level CPU time in inspectIT-ocelot. I googled and found in inspectIT using 'java.lang.management.ThreadMXBean' we can capture 'getCurrentThreadCpuTime' . How do we do it in ocelot. any sample code would really help. do we need to add it in actions?
    Kevin Lotz
    @lotzk

    Hi, im trying to trace an application with ocelot. I need the FQN of each method being called and for this purpose I use the following action

    get_method_fqn:
            input:
              _methodName: String
              _class: Class
            value: "new StringBuilder(_class.getName()).append('.').append(_methodName).toString()"

    However, this does not seem to register constructor calls. Is there a way to trace object creation / constructor calls?

    Patrick-Eichhorn
    @Patrick-Eichhorn

    Hi Kevin, yes, this is possible, but this is not defined in the actions but under the scopes. There you can define if it is a constructor method: with the attribute "is-constructor: true" (see also https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/scopes#method-matcher).

    What does the scope you defined look like?

    Kevin Lotz
    @lotzk

    Hi Patrick, thanks for replying. The scope I'm using looks like this:

      instrumentation:
        scopes:
          allClasses:
            type:
              name: net.ex
              matcher-mode: STARTS_WITH_IGNORE_CASE
    #        methods:
    #          - is-constructor: true

    This works fine als long as the last two lines are commented out. If I include them, no spans are received at all.

    Patrick-Eichhorn
    @Patrick-Eichhorn

    Hi Kevin,
    your configuration is correct, if the last two lines are commented out, this means that only constructors are instrumented. That no spans arrive may be because the constructors are not called. By the way, the constructors in your configuration are instrumented even if the last two lines are commented, because the scope is on all methods of the class.

    Alternatively, it is possible that the span is overlooked accidentally. For this I have a small example from the Petclinic-Demo*:

    Config.png
    Constructors are displayed as {ClassName}.<init> in the spans. For the petclinic example, which instruments the constructors of the class petDetails*, the span would look like in the following picture:
    Span.png
    Malte Hansen
    @Malte-Hansen

    Hi, I am trying to instrument a running JVM process. Running

    java -Dinspectit.config.file-based.path="./" -javaagent:inspectit-ocelot-agent-0.6.jar -jar sampleApplication.jar

    works fine for starting a new application. However, running

    java -jar inspectit-ocelot-agent-0.6.jar 33684 -Dinspectit.config.file-based.path="./"

    whereas 33684 is the PID of the started application does not work. I suspect that the configuration is not loaded correctly. Is there another syntax which I am missing here?

    Patrick-Eichhorn
    @Patrick-Eichhorn
    Hi Malte,
    the problem is: if attaching the agent to a running JVM, the configurations must be set in the process of the application. This is not possible with the java system properties "-Dinspectit", as these only apply to the agent attach process itself.
    You can submit the configuration with the "json" syntax at the end of the command:
    java -jar inspection-ocelot-agent-0.6.jar 33684 '{inspectit:{config:{file-based:{path:"./"}}}}'.
    Please note that the relative path needs to be specified from the application, not from the agent.
    Malte Hansen
    @Malte-Hansen
    Hi Patrick,
    thanks a lot for the advice. With the addition of escaping the double quotation marks within the JSON string on a Windows machine, I managed to monitor a running application as expected. Great!
    Bernd Albersmann
    @albersma69_twitter

    Hi,
    is there any way to tell the inspectIt agent to trace on interfaces instead of implementing classes? I have got my own Spring Classes, which are only interfaces and annotated with "@Repository" as normal procedure for Spring Repositories. The implementation is added by Spring on the fly during compile time / runtime:

    @Repository
    public interface UserAgreementsRepository extends CrudRepository<UserAgreementsEntity, String> {
    
    <UserAgreementsEntity> findByUserId(@NotNull final String teacherUserId);
    }

    I tried all kinds of scope definition, on annotation, on package/class patterns on extending super class pattrens etc, but nothing works. It seems, that interfaces are ignored at all by the ocelot agent. I would like to see, how long f.e. this find-method takes to complete.
    Thanks!

    Marius Oehler
    @mariusoe

    Hi @albersma69_twitter
    a scope for targetting the repository interface could look like this:

    inspectit:
      instrumentation:
        scopes:
          's_spring_repository':
            interfaces:
              - name: 'your.package.UserAgreementsRepository'
            advanced:
              instrument-only-inherited-methods: true

    this will result in an instrumentation of all the methods defined by your interface.

    If you want to target all methods of the implementing class, you can dismiss the advanced property of the example
    Bernd Albersmann
    @albersma69_twitter
    image.png
    Hi Marius,
    thank you. It works "somewhat", but not really satisfactory. The result is one new span inside the trace, which is called
    "$proxy201.findbyteacherid". The method name is correct, the class name not. I'd like to see the interface name in clear words instead of an anonymous proxy201, which implements that interface.
    Where can I find more advice /example on the inspectIt instrumentation grammar other than the little info, which I can find on https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/instrumentation? Thank you!
    Bernd Albersmann
    @albersma69_twitter
    How would I instrument all Repository Interfaces in my app across all packages, whose interace name ends with "Repository"? I tried this, but it does not work:
          'scope_KDO_repository':
            interfaces:
              - name: 'Repository'
                matcher-mode: 'ENDS_WITH'
            advanced:
              instrument-only-inherited-methods: true
    Marius Oehler
    @mariusoe

    Hi @albersma69_twitter

    yes, your approach for instrumenting all repositories whose name ends with the specified term is correct and should work.
    Note that the instrumentation is done asynchronously after the class was loaded. This means that the instrumentation is just active after some delay, thus, it may be the case that some of the spans are missing during this time.

    Regarding the span naming - this is actually not a wrong class name. The class is in fact named $proxy201 because it is generated by Spring under the hood. Due to the fact that the class name of the instrumented method is used as span name by default, your span is getting this name.
    Naming the span according to the interface is not trivial. Nevertheless it is possible. You can have a look on the following configuration where Spring repositories have been traced as well and the resulting spans named according to the defined interface: https://github.com/inspectIT/inspectit-ocelot-configurations/blob/master/extensions/spring-data-repository/spring-data-repository-profiling.yml
    In this case, the name is extracted from the proxy's arguments via Reflection.

    You can find more information on the instrumentation under the "Scopes" and "Rules & Actions" section in the documentation:

    Zhang Zhao
    @zzhao2010

    Hi,
    I ran the demo application provided in the inspectIT/Ocelot documentation (Prometheus and Jaeger Scenario) on my laptop. However after I issue the docker-compose command docker-compose -f docker-compose-prometheus-jaeger.yml up, the demo application was not running properly.. I saw this error message in the log:
    Caused by: com.netflix.client.ClientException: Load balancer does not have available server for client: customers-service
    api-gateway | 2021-05-28 00:10:38.460 WARN 1 --- [nio-8080-exec-4] o.s.c.n.z.filters.post.SendErrorFilter : Error during filtering
    api-gateway | com.netflix.zuul.exception.ZuulException: Forwarding error

    Anyone can help me out here? Thanks!

    Patrick-Eichhorn
    @Patrick-Eichhorn
    Hi @zzhao2010 . One possibility could be, that the docker container "customer-service" is not available (e.g. due to low memory or the service starts slower than the others). You can check this with the docker command docker ps after running docker-compose.
    Zhang Zhao
    @zzhao2010
    @Patrick-Eichhorn How do I fix this issue? I allocated 6 CPUs, 8GB Mem, and 2 GB Swap resources for Docker, which I think is more than the requirement in the doc.
    2 replies
    @Patrick-Eichhorn, yes.. I don’t see any service containers available..
    ❯ docker ps
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    e752bff17c71 openapm/spring-petclinic-api-gateway:inspectit-oce-demo "sh -c ' ./dockerize…" About a minute ago Up About a minute 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9094->9094/tcp, :::9094->9094/tcp, 8081/tcp api-gateway
    972983bf2ce9 openapm/spring-petclinic-discovery-server:inspectit-oce-demo "sh -c ' ./dockerize…" About a minute ago Up About a minute 0.0.0.0:8761->8761/tcp, :::8761->8761/tcp, 0.0.0.0:9095->9095/tcp, :::9095->9095/tcp discovery-server
    159725ed24ff openapm/spring-petclinic-config-server:inspectit-oce-demo "sh -c ' java -XX:+U…" 2 minutes ago Up About a minute 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp, 0.0.0.0:9096->9096/tcp, :::9096->9096/tcp config-server
    dea34d47cfd3 prom/prometheus:v2.7.1 "/bin/prometheus --c…" 2 minutes ago Up 2 minutes 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
    51779eba3eda inspectit/grafana:servicegraph-v2.1 "/run.sh" 2 minutes ago Up 2 minutes 0.0.0.0:3001->3000/tcp, :::3001->3000/tcp grafana
    73f6e4d31837 inspectit/inspectit-ocelot-agent:latest "sh /entrypoint.sh" 2 minutes ago Up 2 minutes agent
    18ab7aad2262 inspectit/spring-petclinic-load:1.0 "/artillery/docker-e…" 2 minutes ago Up 2 minutes load-generator
    22d2f3f7ded5 inspectit/inspectit-ocelot-configurationserver:latest "sh /entrypoint.sh" 2 minutes ago Up 2 minutes 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp ocelot-config-server
    ddd150c26958 jaegertracing/all-in-one:1.6 "/go/bin/standalone-…" 2 minutes ago Up 2 minutes 5775/udp, 5778/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp, :::16686->16686/tcp jaeger
    Alexander Krause
    @Alexander-Krause

    Hello,

    I use Ocelot to monitor the Spring Petclinic. My instrumentation file looks like this:

    inspectit:
    
      tags:
        extra:
          landscape_token: '36efef0d-30a4-48b9-a20b-a1735c2fdc53'
          token_secret: 'ilZsUSzJTJQPCmyk'
          application_name: 'petclinic-demo'
          application_instance_id: 0
          application_language: 'java'
    
      tracing:
        add-common-tags: ALWAYS
    
      metrics:
        enabled: false
    
      instrumentation:
        scopes:
          s_allClasses:
            type: 
              name: org.springframework
              matcher-mode: STARTS_WITH_IGNORE_CASE
    
        rules:
          r_all_methods_no_input_return:
            scopes:
              s_allClasses: true
            tracing:
              start-span: true
              attributes:
                java.fqn: method_fqn_attr
            entry:
              method_fqn_attr:
                action: a_get_method_fqn
    
        actions:
          a_get_method_fqn:
            input:
              _methodName: String
              _class: Class
            value: "new StringBuilder(_class.getName()).append('.').append(_methodName).toString()"
    
      logging:
        debug: true
        trace: false
    
      exporters:
        tracing:
          open-census-agent:
            address: "oc-collector:55678"
            service-name: "OC Agent Local"
            use-insecure: true

    Now, when using a scope broader than org.springframework.samples, i.e., org.springframework or org my analysis application does not receive data at all.

    Can you tell my why?

    Best regards,
    Alex

    Alexander Krause
    @Alexander-Krause

    And after a few minutes the petclinic cannot be reached at all and the log states an exception:

    Exception in thread "http-nio-8080-exec-2" java.lang.ClassCircularityError: rocks/inspectit/ocelot/core/instrumentation/context/InspectitContextImpl$1
        at rocks.inspectit.ocelot.core.instrumentation.context.ContextManager.enterNewContext(ContextManager.java:65)
        at rocks.inspectit.ocelot.core.instrumentation.hook.MethodHook.onEnter(MethodHook.java:80)
        at org.springframework.boot.loader.jar.Handler.parseURL(Handler.java:183)
        at java.base/java.net.URL.<init>(URL.java:674)
        at java.base/java.net.URL.<init>(URL.java:541)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader.checkResource(URLClassPath.java:848)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader.getResource(URLClassPath.java:931)
        at java.base/jdk.internal.loader.URLClassPath.getResource(URLClassPath.java:314)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:455)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at rocks.inspectit.ocelot.core.instrumentation.context.ContextManager.enterNewContext(ContextManager.java:65)
        at rocks.inspectit.ocelot.core.instrumentation.hook.MethodHook.onEnter(MethodHook.java:80)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:123)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:204)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.bestEffortLoadClass(PackagingDataCalculator.java:228)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.computeBySTEP(PackagingDataCalculator.java:135)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(PackagingDataCalculator.java:100)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.calculate(PackagingDataCalculator.java:58)
        at ch.qos.logback.classic.spi.ThrowableProxy.calculatePackagingData(ThrowableProxy.java:142)
        at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:122)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.log(Logger.java:765)
        at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
        at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
        at java.logging/java.util.logging.Logger.log(Logger.java:979)
        at java.logging/java.util.logging.Logger.doLog(Logger.java:1006)
        at java.logging/java.util.logging.Logger.logp(Logger.java:1283)
        at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:175)
        at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:141)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1725)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

    Again, this seems to be only the case when using a "broad" instrumentation scope. Its all running in Docker on MacOs with enough resources.

    Best regards,
    Alex

    1 reply
    kaushalshriyan
    @kaushalshriyan
    Hi All,
    Márton Szücs
    @martonsz
    Hi!
    How can I disable metrics for http_in_count? I get thousands of individual metrics for this because our API uses the path as a parameter.
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg1,...}
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg2,...}
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg3,...}
    ... thousands!
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/argN,...}
    1 reply
    I am using kubernetes configuration object
    # exampel for JMX
    ...
      metrics:
        jmx:
          enabled: true
          lower-case-metric-name: true
          object-names:
            # allow-listed (true) and deny-listed (false) object names
            '[Catalina:type=Connector,port=8080]': true
            '[Catalina:type=GlobalRequestProcessor,name="http-nio-8080"]': true
            '[Catalina:type=Manager,host=localhost,*]': true
    ...
    Matthias Huber
    @matthias-huber

    Hi, I am using the inspectIT Ocelot Standard configuration + a view to capture min and max values for HTTP In and out calls
    `inspectit:
    metrics:
    definitions:

      '[http/in/responsetime]':
        unit: ms
        views:
          '[http/in/responsetime]':
            aggregation: QUANTILES
            quantiles: [0, 1]
            tags: 
              'http_path': true
              'http_status': true
              'http_method': true
              'error': true
    
      '[http/out/responsetime]':
        unit: ms
        views:
          '[http/out/responsetime]':
            aggregation: QUANTILES
            quantiles: [0, 1]
            tags:
              'http_host': true
              'http_path': true
              'http_status': true
              'http_method': true
              'error': true

    `
    I get metrics but the average (calculated by sum/ count) is lower as the min value. Min and Max values seem to be reasonable (have to re-check by looking at Tracing Data).

    4 replies
    inspectit_avg_min_max.png
    Is my configuration of the views incorrect? Or do you have any idea what causes the difference, e.g. wrong PromQL query like mixing of different aggregation windows?
    Alexander Krause
    @Alexander-Krause

    Hello, the OpenTelemetry collector includes an OpenCensus receiver that seems to use this protobuf version for traces.

    I tried to send OC traces from Ocelot to the OpenTelemetry collector, but no data was received. I know that the OpenTelemetry support is still in development in Ocelot, but I wanted to try the backwards compatability of OC for the moment. Currently, I think that Ocelot uses a different protobuf version than the receiver, but couldn't figure out the version number.

    Did you successfully send traces from Ocelot to the OpenTelemetry collector in testing? Or is the protobuf version really in issue here?

    1 reply
    Matthias Huber
    @matthias-huber
    Hello guys,
    is it true that if I disable tracing with "-Dinspectit.tracing.enabled=false" the agent will not gather any HTTP Statistics (metrics)? Furthermore, is it true that if I set sampling_probalitity < 1.0 that also the HTTP Statistics are not covering all 100% of requests? In other words: Are HTTP Statistics only gathered if a request is actually traced?
    8 replies
    Matthias Huber
    @matthias-huber

    Hello guys,
    I have another situation that I don't know if my assumptions are true and if they are true I don't know how to deal with it. So what I try to solve is to extend the outgoing http metrics with another label that I need to distinguish calls to the same backend. Therefor I extend the available metric definition and set the value via the rule 'r_http_client_record_metric' as data-tag. I extract the tag somewhere during request processing (use up+down progagation before the backend call is initiated. In total this works. But I have an edge case after the initial start of the application where the label is empty for the first call (maybe more if they are executed fast after another -> have not tried that).

    I had a look at the instrumentation statistics from the agent before and after the request. The number of instrumented classes increases, probably as the classes involved in sending the request to the backend and also the class of the object the additional tag is extracted from is loaded. So I wonder if the label is not set because of the aynchronous instrumentation process. Meaning until the class hasn't change, the label will be empty. Can you confirm this behaviour? And if so is there any solution to it?

    I already thought of extracting the value deeper in the call stack so at best at the same class/method the other http statistics labels like http_path are extracted. Because they are available for the first call and the instrumentation points are already placed before the first request as the class has already been loaded?

    2 replies
    Bernd Albersmann
    @albersma69_twitter

    Hi guys,
    I'am currently experimenting with your boomerang-opentelemetry.js plugin to export traces / spans to Jaeger. I'am currently facing 2 problems.

    1. Whatever I as libraray name or version into the getTracer method, like this:

      const tracer = window.BOOMR.plugins.OpenTelemetry.getTracer("my-library-name", "v1.0");

      I always get an javascript error :
      Cannot read properties of undefined (reading 'getTracer')

    2. When I don't use opentelemetry collector (by removing the collectorConfiguration section), the plugin tries to send the spans to the ocelot-eum-server. That POST crashes with a NullPointerException inside the EUM-Server:
      java.lang.NullPointerException: null

       at rocks.inspectit.oce.eum.server.rest.TraceController.spans(TraceController.java:58)
       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
       at java.base/java.lang.reflect.Method.invoke(Unknown Source)
       at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
       at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
       at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
       at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
       at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
       at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
       at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
       at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
       at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
       at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
       at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    Thank you!

    3 replies
    Bernd Albersmann
    @albersma69_twitter

    Hi guys,
    I tried to export traces to OpenTelemetryCollector using the OTLP Exporter of InspectIT 2.0.2. But I can't see any tracing entries in the InspectIT console or in the OpenTelemetryCollector or in the Jaeger UI. I used the following environment variables in my docker-compose:

      - INSPECTIT_EXPORTERS_TRACES_OTLP_ENABLED=ENABLED
      - INSPECTIT_EXPORTERS_TRACES_OTLP_ENDPOINT=http://otel-collector:4317
      - INSPECTIT_EXPORTERS_TRACES_OTLP_PROTOCOL=grpc
      - INSPECTIT_EXPORTERS_TRACING_LOGGING_ENABLED=ENABLED
      - INSPECTIT_EXPORTERS_TRACING_SERVICE_NAME=visits-service

    or with http/protobuf

      - INSPECTIT_EXPORTERS_TRACES_OTLP_ENABLED=ENABLED
      - INSPECTIT_EXPORTERS_TRACES_OTLP_ENDPOINT=http://otel-collector:4318
      - INSPECTIT_EXPORTERS_TRACES_OTLP_PROTOCOL=http/protobuf
      - INSPECTIT_EXPORTERS_TRACING_LOGGING_ENABLED=ENABLED
      - INSPECTIT_EXPORTERS_TRACING_SERVICE_NAME=vets-service

    I know that you stated in https://inspectit.github.io/inspectit-ocelot/docs/breaking-changes/Breaking%20Changes , that auto tracing is not available. Does that mean, that trace exporting to OtelCollector is not running at all? Or do I have to provide special instrumentation files via ocelot configuration server?
    Thanks!

    6 replies
    Bernd Albersmann
    @albersma69_twitter
    @albersma69_twitter @albersma69_twitter
    Hi again,
    I'm trying to get inspetit-Agent running with an openliberty-Server, but it does not produce the expected output. No metrics at all, traces appear only partially.
    To attach inspectit to openliberty server and expose metrics and traces I created the following jvm.options file:

    -javaagent:/inspectit-ocelot-agent.jar
    -Dinspectit.exporters.metrics.prometheus.enabled=ENABLED
    -Dinspectit.exporters.metrics.prometheus.port=9095
    -Dinspectit.exporters.tracing.otlp.enabled=ENABLED
    -Dinspectit.exporters.tracing.otlp.endpoint=http://collector:4317
    -Dinspectit.exporters.tracing.otlp.protocol=grpc
    -Dinspectit.exporters.tracing.logging.enabled=ENABLED
    -Dinspectit.exporters.tracing.service-name=openliberty-tracing
    -Dinspectit.config.http.enabled=true
    -Dinspectit.config.http.url=http://ocelot-config-server:8090/api/v1/agent/configuration
    -Dinspectit.logging.file.enabled=true
    -Dinspectit.logging.file.path=/tmp/inspectit
    -Dinspectit-name=openliberty

    After starting openliberty with the inspectit agent, I do not see any http_in or http_out metrics. But I do see jvm and gc-metrics. Furthermore I don't see any REST-Endpoints as traces / spans (Such as GET.., PUT ..., POST...), but only those ones in the JavaCode, where I added manual tracing to. Even if I add metrics to methods I don't see those metrics appear. I tried all variants, with and without manual instrumentation, inspectit 2.03. and 1.16.0, only metrics exporter and so on, but no metrics and only self-instrumented spans. The agent log file is somewhat difficult to interpret for me, so hopefully you can do better:

    aunching defaultServer (Open Liberty 22.0.0.6/wlp-1.0.65.cl220620220523-1607) on Eclipse OpenJ9 VM, version 11.0.15+10 (en_US)
    [AUDIT ] CWWKE0001I: The server defaultServer has been launched.
    [AUDIT ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
    [AUDIT ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
    2022-06-29 15:41:51,753 INFO 119 --- [inspectIT] [ Thread-0] rocks.inspectit.ocelot.core.AgentImpl : Starting inspectIT Ocelot Agent...
    2022-06-29 15:41:51,755 INFO 121 --- [inspectIT] [ Thread-0] rocks.inspectit.ocelot.core.AgentImpl : Version: 2.0.3
    2022-06-29 15:41:51,756 INFO 122 --- [inspectIT] [ Thread-0] rocks.inspectit.ocelot.core.AgentImpl : Build Date: Tue Jun 21 13:44:15 UTC 2022
    2022-06-29 15:41:51,756 INFO 122 --- [inspectIT] [ Thread-0] rocks.inspectit.ocelot.core.AgentImpl : OpenTelemetry was loaded in inspectIT classloader
    [AUDIT ] CWWKZ0058I: Monitoring dropins for applications.
    2022-06-29 15:41:52,466 INFO 832 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : Initializing HTTP based configuration from URL: http://ocelot-config-server:8090/api/v1/agent/configuration?id=1%404623bd3975dc&service=InspectIT+Agent
    2022-06-29 15:41:52,809 INFO 1175 --- [inspectIT] [ Thread-0] r.i.o.c.c.p.h.HttpPropertySourceState : HTTP Configuration has successfully been fetched.
    2022-06-29 15:41:52,812 ERROR 1178 --- [inspectIT] [ Thread-0] r.i.o.c.c.p.h.HttpPropertySourceState : Could not write persistence file for HTTP-configuration.
    java.nio.file.AccessDeniedException: /InspectIT Agent
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
    .....
    1 reply
    second part of agent log file:
    AUDIT ] CWWKT0016I: Web application available (default_host): http://4623bd3975dc:9080/metrics/
    [AUDIT ] CWWKT0016I: Web application available (default_host): http://4623bd3975dc:9080/ibm/api/
    2022-06-29 15:41:52,834 INFO 1200 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : Registered Configuration Sources:
    2022-06-29 15:41:52,835 INFO 1201 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : systemProperties
    2022-06-29 15:41:52,835 INFO 1201 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : systemEnvironment
    2022-06-29 15:41:52,836 INFO 1202 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : httpBasedConfig
    2022-06-29 15:41:52,836 INFO 1202 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : inspectitDefaults
    2022-06-29 15:41:52,836 INFO 1202 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : inspectitEnvironment
    [AUDIT ] CWWKT0016I: Web application available (default_host): http://4623bd3975dc:9080/
    [AUDIT ] CWWKZ0001I: Application guide-getting-started started in 0.872 seconds.
    [AUDIT ] CWWKF0012I: The server installed the following features: [cdi-3.0, concurrent-2.0, distributedMap-1.0, jndi-1.0, json-1.0, jsonb-2.0, jsonp-2.0, monitor-1.0, mpConfig-3.0, mpMetrics-4.0, restfulWS-3.0, restfulWSClient-3.0, servlet-5.0, ssl-1.0, transportSecurity-1.0].
    [AUDIT ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 2.306 seconds.
    2022-06-29 15:41:54,928 INFO 3294 --- [inspectIT] [ Thread-0] r.i.o.c.o.trace.CustomIdGenerator : Use of trace IDs with the default length (128 bits).
    2022-06-29 15:41:55,726 INFO 4092 --- [inspectIT] [ Thread-0] r.i.o.c.i.c.ContextPropagationUtil : Using B3 format for context propagation.
    2022-06-29 15:41:55,798 INFO 4164 --- [inspectIT] [ Thread-0] o.c.c.p.f.ConfigurationDirectoriesPoller : Starting config directory polling service...
    2022-06-29 15:41:55,809 INFO 4175 --- [inspectIT] [ Thread-0] o.c.c.p.f.ConfigurationDirectoriesPoller : No directories to watch registered, terminating poll service.
    2022-06-29 15:41:55,810 INFO 4176 --- [inspectIT] [ Thread-0] r.i.o.c.c.p.h.HttpConfigurationPoller : Starting HTTP configuration polling service.
    2022-06-29 15:41:55,811 INFO 4177 --- [inspectIT] [ Thread-0] r.i.o.c.o.OpenTelemetryControllerImpl : The spanExporter io.opentelemetry.exporter.logging.LoggingSpanExporter for the service LoggingTraceExporterService was successfully registered.
    2022-06-29 15:41:55,812 INFO 4178 --- [inspectIT] [ Thread-0] r.i.o.c.o.OpenTelemetryControllerImpl : The service LoggingTraceExporterService was successfully registered.
    2022-06-29 15:41:55,813 INFO 4179 --- [inspectIT] [ Thread-0] r.i.o.c.e.LoggingTraceExporterService : Starting LoggingTraceExporterService
    2022-06-29 15:41:55,931 INFO 4297 --- [inspectIT] [ Thread-0] r.i.o.c.o.OpenTelemetryControllerImpl : The spanExporter io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter for the service OtlpTraceExporterService was successfully registered.
    2022-06-29 15:41:55,932 INFO 4298 --- [inspectIT] [ Thread-0] r.i.o.c.o.OpenTelemetryControllerImpl : The service OtlpTraceExporterService was successfully registered.
    2022-06-29 15:41:55,932 INFO 4298 --- [inspectIT] [ Thread-0] r.i.o.c.e.OtlpTraceExporterService : Starting OTLP Trace Exporter with endpoint http://collector:4317
    2022-06-29 15:41:55,933 INFO 4299 --- [inspectIT] [ Thread-0] r.i.o.c.o.OpenTelemetryControllerImpl : The service PrometheusExporterService was successfully registered.
    2022-06-29 15:41:55,934 INFO 4300 --- [inspectIT] [ Thread-0] r.i.o.c.e.PrometheusExporterService : Starting Prometheus Exporter on 0.0.0.0:9095
    2022-06-29 15:41:55,935 INFO 4301 --- [inspectIT] [ Thread-0] i.o.c.m.s.AbstractPollingM
    Bernd Albersmann
    @albersma69_twitter
    As far as I know, openliberty is built on OSGi, which has its own class loading mechanism, different from normal java application. It is possible that InspectIT does not support OSGI-class loading?