Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Patrick-Eichhorn
    @Patrick-Eichhorn

    Hi Kevin, yes, this is possible, but this is not defined in the actions but under the scopes. There you can define if it is a constructor method: with the attribute "is-constructor: true" (see also https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/scopes#method-matcher).

    What does the scope you defined look like?

    Kevin Lotz
    @lotzk

    Hi Patrick, thanks for replying. The scope I'm using looks like this:

      instrumentation:
        scopes:
          allClasses:
            type:
              name: net.ex
              matcher-mode: STARTS_WITH_IGNORE_CASE
    #        methods:
    #          - is-constructor: true

    This works fine als long as the last two lines are commented out. If I include them, no spans are received at all.

    Patrick-Eichhorn
    @Patrick-Eichhorn

    Hi Kevin,
    your configuration is correct, if the last two lines are commented out, this means that only constructors are instrumented. That no spans arrive may be because the constructors are not called. By the way, the constructors in your configuration are instrumented even if the last two lines are commented, because the scope is on all methods of the class.

    Alternatively, it is possible that the span is overlooked accidentally. For this I have a small example from the Petclinic-Demo*:

    Config.png
    Constructors are displayed as {ClassName}.<init> in the spans. For the petclinic example, which instruments the constructors of the class petDetails*, the span would look like in the following picture:
    Span.png
    Malte Hansen
    @Malte-Hansen

    Hi, I am trying to instrument a running JVM process. Running

    java -Dinspectit.config.file-based.path="./" -javaagent:inspectit-ocelot-agent-0.6.jar -jar sampleApplication.jar

    works fine for starting a new application. However, running

    java -jar inspectit-ocelot-agent-0.6.jar 33684 -Dinspectit.config.file-based.path="./"

    whereas 33684 is the PID of the started application does not work. I suspect that the configuration is not loaded correctly. Is there another syntax which I am missing here?

    Patrick-Eichhorn
    @Patrick-Eichhorn
    Hi Malte,
    the problem is: if attaching the agent to a running JVM, the configurations must be set in the process of the application. This is not possible with the java system properties "-Dinspectit", as these only apply to the agent attach process itself.
    You can submit the configuration with the "json" syntax at the end of the command:
    java -jar inspection-ocelot-agent-0.6.jar 33684 '{inspectit:{config:{file-based:{path:"./"}}}}'.
    Please note that the relative path needs to be specified from the application, not from the agent.
    Malte Hansen
    @Malte-Hansen
    Hi Patrick,
    thanks a lot for the advice. With the addition of escaping the double quotation marks within the JSON string on a Windows machine, I managed to monitor a running application as expected. Great!
    Bernd Albersmann
    @albersma69_twitter

    Hi,
    is there any way to tell the inspectIt agent to trace on interfaces instead of implementing classes? I have got my own Spring Classes, which are only interfaces and annotated with "@Repository" as normal procedure for Spring Repositories. The implementation is added by Spring on the fly during compile time / runtime:

    @Repository
    public interface UserAgreementsRepository extends CrudRepository<UserAgreementsEntity, String> {
    
    <UserAgreementsEntity> findByUserId(@NotNull final String teacherUserId);
    }

    I tried all kinds of scope definition, on annotation, on package/class patterns on extending super class pattrens etc, but nothing works. It seems, that interfaces are ignored at all by the ocelot agent. I would like to see, how long f.e. this find-method takes to complete.
    Thanks!

    Marius Oehler
    @mariusoe

    Hi @albersma69_twitter
    a scope for targetting the repository interface could look like this:

    inspectit:
      instrumentation:
        scopes:
          's_spring_repository':
            interfaces:
              - name: 'your.package.UserAgreementsRepository'
            advanced:
              instrument-only-inherited-methods: true

    this will result in an instrumentation of all the methods defined by your interface.

    If you want to target all methods of the implementing class, you can dismiss the advanced property of the example
    Bernd Albersmann
    @albersma69_twitter
    image.png
    Hi Marius,
    thank you. It works "somewhat", but not really satisfactory. The result is one new span inside the trace, which is called
    "$proxy201.findbyteacherid". The method name is correct, the class name not. I'd like to see the interface name in clear words instead of an anonymous proxy201, which implements that interface.
    Where can I find more advice /example on the inspectIt instrumentation grammar other than the little info, which I can find on https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/instrumentation? Thank you!
    Bernd Albersmann
    @albersma69_twitter
    How would I instrument all Repository Interfaces in my app across all packages, whose interace name ends with "Repository"? I tried this, but it does not work:
          'scope_KDO_repository':
            interfaces:
              - name: 'Repository'
                matcher-mode: 'ENDS_WITH'
            advanced:
              instrument-only-inherited-methods: true
    Marius Oehler
    @mariusoe

    Hi @albersma69_twitter

    yes, your approach for instrumenting all repositories whose name ends with the specified term is correct and should work.
    Note that the instrumentation is done asynchronously after the class was loaded. This means that the instrumentation is just active after some delay, thus, it may be the case that some of the spans are missing during this time.

    Regarding the span naming - this is actually not a wrong class name. The class is in fact named $proxy201 because it is generated by Spring under the hood. Due to the fact that the class name of the instrumented method is used as span name by default, your span is getting this name.
    Naming the span according to the interface is not trivial. Nevertheless it is possible. You can have a look on the following configuration where Spring repositories have been traced as well and the resulting spans named according to the defined interface: https://github.com/inspectIT/inspectit-ocelot-configurations/blob/master/extensions/spring-data-repository/spring-data-repository-profiling.yml
    In this case, the name is extracted from the proxy's arguments via Reflection.

    You can find more information on the instrumentation under the "Scopes" and "Rules & Actions" section in the documentation:

    Zhang Zhao
    @zzhao2010

    Hi,
    I ran the demo application provided in the inspectIT/Ocelot documentation (Prometheus and Jaeger Scenario) on my laptop. However after I issue the docker-compose command docker-compose -f docker-compose-prometheus-jaeger.yml up, the demo application was not running properly.. I saw this error message in the log:
    Caused by: com.netflix.client.ClientException: Load balancer does not have available server for client: customers-service
    api-gateway | 2021-05-28 00:10:38.460 WARN 1 --- [nio-8080-exec-4] o.s.c.n.z.filters.post.SendErrorFilter : Error during filtering
    api-gateway | com.netflix.zuul.exception.ZuulException: Forwarding error

    Anyone can help me out here? Thanks!

    Patrick-Eichhorn
    @Patrick-Eichhorn
    Hi @zzhao2010 . One possibility could be, that the docker container "customer-service" is not available (e.g. due to low memory or the service starts slower than the others). You can check this with the docker command docker ps after running docker-compose.
    Zhang Zhao
    @zzhao2010
    @Patrick-Eichhorn How do I fix this issue? I allocated 6 CPUs, 8GB Mem, and 2 GB Swap resources for Docker, which I think is more than the requirement in the doc.
    2 replies
    @Patrick-Eichhorn, yes.. I don’t see any service containers available..
    ❯ docker ps
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    e752bff17c71 openapm/spring-petclinic-api-gateway:inspectit-oce-demo "sh -c ' ./dockerize…" About a minute ago Up About a minute 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9094->9094/tcp, :::9094->9094/tcp, 8081/tcp api-gateway
    972983bf2ce9 openapm/spring-petclinic-discovery-server:inspectit-oce-demo "sh -c ' ./dockerize…" About a minute ago Up About a minute 0.0.0.0:8761->8761/tcp, :::8761->8761/tcp, 0.0.0.0:9095->9095/tcp, :::9095->9095/tcp discovery-server
    159725ed24ff openapm/spring-petclinic-config-server:inspectit-oce-demo "sh -c ' java -XX:+U…" 2 minutes ago Up About a minute 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp, 0.0.0.0:9096->9096/tcp, :::9096->9096/tcp config-server
    dea34d47cfd3 prom/prometheus:v2.7.1 "/bin/prometheus --c…" 2 minutes ago Up 2 minutes 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
    51779eba3eda inspectit/grafana:servicegraph-v2.1 "/run.sh" 2 minutes ago Up 2 minutes 0.0.0.0:3001->3000/tcp, :::3001->3000/tcp grafana
    73f6e4d31837 inspectit/inspectit-ocelot-agent:latest "sh /entrypoint.sh" 2 minutes ago Up 2 minutes agent
    18ab7aad2262 inspectit/spring-petclinic-load:1.0 "/artillery/docker-e…" 2 minutes ago Up 2 minutes load-generator
    22d2f3f7ded5 inspectit/inspectit-ocelot-configurationserver:latest "sh /entrypoint.sh" 2 minutes ago Up 2 minutes 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp ocelot-config-server
    ddd150c26958 jaegertracing/all-in-one:1.6 "/go/bin/standalone-…" 2 minutes ago Up 2 minutes 5775/udp, 5778/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp, :::16686->16686/tcp jaeger
    Alexander Krause
    @Alexander-Krause

    Hello,

    I use Ocelot to monitor the Spring Petclinic. My instrumentation file looks like this:

    inspectit:
    
      tags:
        extra:
          landscape_token: '36efef0d-30a4-48b9-a20b-a1735c2fdc53'
          token_secret: 'ilZsUSzJTJQPCmyk'
          application_name: 'petclinic-demo'
          application_instance_id: 0
          application_language: 'java'
    
      tracing:
        add-common-tags: ALWAYS
    
      metrics:
        enabled: false
    
      instrumentation:
        scopes:
          s_allClasses:
            type: 
              name: org.springframework
              matcher-mode: STARTS_WITH_IGNORE_CASE
    
        rules:
          r_all_methods_no_input_return:
            scopes:
              s_allClasses: true
            tracing:
              start-span: true
              attributes:
                java.fqn: method_fqn_attr
            entry:
              method_fqn_attr:
                action: a_get_method_fqn
    
        actions:
          a_get_method_fqn:
            input:
              _methodName: String
              _class: Class
            value: "new StringBuilder(_class.getName()).append('.').append(_methodName).toString()"
    
      logging:
        debug: true
        trace: false
    
      exporters:
        tracing:
          open-census-agent:
            address: "oc-collector:55678"
            service-name: "OC Agent Local"
            use-insecure: true

    Now, when using a scope broader than org.springframework.samples, i.e., org.springframework or org my analysis application does not receive data at all.

    Can you tell my why?

    Best regards,
    Alex

    Alexander Krause
    @Alexander-Krause

    And after a few minutes the petclinic cannot be reached at all and the log states an exception:

    Exception in thread "http-nio-8080-exec-2" java.lang.ClassCircularityError: rocks/inspectit/ocelot/core/instrumentation/context/InspectitContextImpl$1
        at rocks.inspectit.ocelot.core.instrumentation.context.ContextManager.enterNewContext(ContextManager.java:65)
        at rocks.inspectit.ocelot.core.instrumentation.hook.MethodHook.onEnter(MethodHook.java:80)
        at org.springframework.boot.loader.jar.Handler.parseURL(Handler.java:183)
        at java.base/java.net.URL.<init>(URL.java:674)
        at java.base/java.net.URL.<init>(URL.java:541)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader.checkResource(URLClassPath.java:848)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader.getResource(URLClassPath.java:931)
        at java.base/jdk.internal.loader.URLClassPath.getResource(URLClassPath.java:314)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:455)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at rocks.inspectit.ocelot.core.instrumentation.context.ContextManager.enterNewContext(ContextManager.java:65)
        at rocks.inspectit.ocelot.core.instrumentation.hook.MethodHook.onEnter(MethodHook.java:80)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:123)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:204)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.bestEffortLoadClass(PackagingDataCalculator.java:228)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.computeBySTEP(PackagingDataCalculator.java:135)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(PackagingDataCalculator.java:100)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.calculate(PackagingDataCalculator.java:58)
        at ch.qos.logback.classic.spi.ThrowableProxy.calculatePackagingData(ThrowableProxy.java:142)
        at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:122)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.log(Logger.java:765)
        at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
        at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
        at java.logging/java.util.logging.Logger.log(Logger.java:979)
        at java.logging/java.util.logging.Logger.doLog(Logger.java:1006)
        at java.logging/java.util.logging.Logger.logp(Logger.java:1283)
        at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:175)
        at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:141)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1725)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

    Again, this seems to be only the case when using a "broad" instrumentation scope. Its all running in Docker on MacOs with enough resources.

    Best regards,
    Alex

    1 reply
    kaushalshriyan
    @kaushalshriyan
    Hi All,
    Márton Szücs
    @martonsz
    Hi!
    How can I disable metrics for http_in_count? I get thousands of individual metrics for this because our API uses the path as a parameter.
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg1,...}
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg2,...}
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg3,...}
    ... thousands!
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/argN,...}
    1 reply
    I am using kubernetes configuration object
    # exampel for JMX
    ...
      metrics:
        jmx:
          enabled: true
          lower-case-metric-name: true
          object-names:
            # allow-listed (true) and deny-listed (false) object names
            '[Catalina:type=Connector,port=8080]': true
            '[Catalina:type=GlobalRequestProcessor,name="http-nio-8080"]': true
            '[Catalina:type=Manager,host=localhost,*]': true
    ...
    Matthias Huber
    @matthias-huber

    Hi, I am using the inspectIT Ocelot Standard configuration + a view to capture min and max values for HTTP In and out calls
    `inspectit:
    metrics:
    definitions:

      '[http/in/responsetime]':
        unit: ms
        views:
          '[http/in/responsetime]':
            aggregation: QUANTILES
            quantiles: [0, 1]
            tags: 
              'http_path': true
              'http_status': true
              'http_method': true
              'error': true
    
      '[http/out/responsetime]':
        unit: ms
        views:
          '[http/out/responsetime]':
            aggregation: QUANTILES
            quantiles: [0, 1]
            tags:
              'http_host': true
              'http_path': true
              'http_status': true
              'http_method': true
              'error': true

    `
    I get metrics but the average (calculated by sum/ count) is lower as the min value. Min and Max values seem to be reasonable (have to re-check by looking at Tracing Data).

    4 replies
    inspectit_avg_min_max.png
    Is my configuration of the views incorrect? Or do you have any idea what causes the difference, e.g. wrong PromQL query like mixing of different aggregation windows?
    Alexander Krause
    @Alexander-Krause

    Hello, the OpenTelemetry collector includes an OpenCensus receiver that seems to use this protobuf version for traces.

    I tried to send OC traces from Ocelot to the OpenTelemetry collector, but no data was received. I know that the OpenTelemetry support is still in development in Ocelot, but I wanted to try the backwards compatability of OC for the moment. Currently, I think that Ocelot uses a different protobuf version than the receiver, but couldn't figure out the version number.

    Did you successfully send traces from Ocelot to the OpenTelemetry collector in testing? Or is the protobuf version really in issue here?

    1 reply
    Matthias Huber
    @matthias-huber
    Hello guys,
    is it true that if I disable tracing with "-Dinspectit.tracing.enabled=false" the agent will not gather any HTTP Statistics (metrics)? Furthermore, is it true that if I set sampling_probalitity < 1.0 that also the HTTP Statistics are not covering all 100% of requests? In other words: Are HTTP Statistics only gathered if a request is actually traced?
    8 replies
    Matthias Huber
    @matthias-huber

    Hello guys,
    I have another situation that I don't know if my assumptions are true and if they are true I don't know how to deal with it. So what I try to solve is to extend the outgoing http metrics with another label that I need to distinguish calls to the same backend. Therefor I extend the available metric definition and set the value via the rule 'r_http_client_record_metric' as data-tag. I extract the tag somewhere during request processing (use up+down progagation before the backend call is initiated. In total this works. But I have an edge case after the initial start of the application where the label is empty for the first call (maybe more if they are executed fast after another -> have not tried that).

    I had a look at the instrumentation statistics from the agent before and after the request. The number of instrumented classes increases, probably as the classes involved in sending the request to the backend and also the class of the object the additional tag is extracted from is loaded. So I wonder if the label is not set because of the aynchronous instrumentation process. Meaning until the class hasn't change, the label will be empty. Can you confirm this behaviour? And if so is there any solution to it?

    I already thought of extracting the value deeper in the call stack so at best at the same class/method the other http statistics labels like http_path are extracted. Because they are available for the first call and the instrumentation points are already placed before the first request as the class has already been loaded?

    2 replies