Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Jonas Kunz
    @JonasKunz
    You need to instrument the methods of your services, have a look at the following part of the documentation:
    https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/scopes
    tgrperf
    @tgrperf
    I've defined scopes for my business & dao services/transactions, but to which predifined rule i have to attache my scopes? exemple for my oracle sql tracing i've used servicegraph_record_jdbc_calls , which rule to use for my need.
    Jonas Kunz
    @JonasKunz
    There currently is no rule (will soon change), so create one yourself:
    inspectit:
      instrumentation:
        rules:
          trace_service:
            scopes:
              your_scope: true 
            tracing:
              start-span: true
    tgrperf
    @tgrperf
    OK, it's working now with this config, thanks a lot.
    tgrperf
    @tgrperf
    Hello, i'm facing zipkin/elasticsearch disk space issues (high disk consumption), is there a way to limit spans less than 10s for example with inspectit configuration??
    Jonas Kunz
    @JonasKunz
    You have to options to reduce the number of traces:
    cattox
    @cattox
    Hi all, I'm trying to do something that I believe is quite simple, but I'm struggling to find the correct configuration for it. I've spent some time reading the documentation and searching the web with little success. I have my scenario working as I wanted except for a little detail. I want to consider only the spans that take more than 10ms to complete. Can you point me out the correct way to achieve it? Thanks in advance.
    cattox
    @cattox
    I know that the previous message talks about keeping only slow traces, but what I really want is to keep all traces only with slow spans.
    Jonas Kunz
    @JonasKunz

    Unfortunately, here the problem is the same: With OpenCensus / OpenTelemetry / OpenTracing, a span needs to know whether or not it shall be sampled at thepoint in time when it is started.

    Why is this necessary?
    When a span is started, it makes itself the parent of all following spans. Therefore it already needs to know at this point, if it should make itself the parent of following spans or not.

    RRadziejewski
    @RRadziejewski
    How to add some tracing config which can be assigned to every project so i can basically test if traces are working, Im on this from 2 days, please help
    @JonasKunz Can you please help ?
    Jonas Kunz
    @JonasKunz
    As stated in git, by using the configuration provided with the demo you should be able to see traces.
    Alternatively yo ucan wait until we release version 1.3 this week, which will include the tracing configuration in it's core. This means then no additional configuration except for the jaeger exporter is required to see HTTP traces.
    Jonas Kunz
    @JonasKunz
    *in github
    RRadziejewski
    @RRadziejewski
    When will be the release
    ?
    unknownuu
    @unknownuu
    How do we get method level CPU time in inspectIT-ocelot. I googled and found in inspectIT using 'java.lang.management.ThreadMXBean' we can capture 'getCurrentThreadCpuTime' . How do we do it in ocelot. any sample code would really help. do we need to add it in actions?
    Kevin Lotz
    @lotzk

    Hi, im trying to trace an application with ocelot. I need the FQN of each method being called and for this purpose I use the following action

    get_method_fqn:
            input:
              _methodName: String
              _class: Class
            value: "new StringBuilder(_class.getName()).append('.').append(_methodName).toString()"

    However, this does not seem to register constructor calls. Is there a way to trace object creation / constructor calls?

    Patrick-Eichhorn
    @Patrick-Eichhorn

    Hi Kevin, yes, this is possible, but this is not defined in the actions but under the scopes. There you can define if it is a constructor method: with the attribute "is-constructor: true" (see also https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/scopes#method-matcher).

    What does the scope you defined look like?

    Kevin Lotz
    @lotzk

    Hi Patrick, thanks for replying. The scope I'm using looks like this:

      instrumentation:
        scopes:
          allClasses:
            type:
              name: net.ex
              matcher-mode: STARTS_WITH_IGNORE_CASE
    #        methods:
    #          - is-constructor: true

    This works fine als long as the last two lines are commented out. If I include them, no spans are received at all.

    Patrick-Eichhorn
    @Patrick-Eichhorn

    Hi Kevin,
    your configuration is correct, if the last two lines are commented out, this means that only constructors are instrumented. That no spans arrive may be because the constructors are not called. By the way, the constructors in your configuration are instrumented even if the last two lines are commented, because the scope is on all methods of the class.

    Alternatively, it is possible that the span is overlooked accidentally. For this I have a small example from the Petclinic-Demo*:

    Config.png
    Constructors are displayed as {ClassName}.<init> in the spans. For the petclinic example, which instruments the constructors of the class petDetails*, the span would look like in the following picture:
    Span.png
    Malte Hansen
    @Malte-Hansen

    Hi, I am trying to instrument a running JVM process. Running

    java -Dinspectit.config.file-based.path="./" -javaagent:inspectit-ocelot-agent-0.6.jar -jar sampleApplication.jar

    works fine for starting a new application. However, running

    java -jar inspectit-ocelot-agent-0.6.jar 33684 -Dinspectit.config.file-based.path="./"

    whereas 33684 is the PID of the started application does not work. I suspect that the configuration is not loaded correctly. Is there another syntax which I am missing here?

    Patrick-Eichhorn
    @Patrick-Eichhorn
    Hi Malte,
    the problem is: if attaching the agent to a running JVM, the configurations must be set in the process of the application. This is not possible with the java system properties "-Dinspectit", as these only apply to the agent attach process itself.
    You can submit the configuration with the "json" syntax at the end of the command:
    java -jar inspection-ocelot-agent-0.6.jar 33684 '{inspectit:{config:{file-based:{path:"./"}}}}'.
    Please note that the relative path needs to be specified from the application, not from the agent.
    Malte Hansen
    @Malte-Hansen
    Hi Patrick,
    thanks a lot for the advice. With the addition of escaping the double quotation marks within the JSON string on a Windows machine, I managed to monitor a running application as expected. Great!
    Bernd Albersmann
    @albersma69_twitter

    Hi,
    is there any way to tell the inspectIt agent to trace on interfaces instead of implementing classes? I have got my own Spring Classes, which are only interfaces and annotated with "@Repository" as normal procedure for Spring Repositories. The implementation is added by Spring on the fly during compile time / runtime:

    @Repository
    public interface UserAgreementsRepository extends CrudRepository<UserAgreementsEntity, String> {
    
    <UserAgreementsEntity> findByUserId(@NotNull final String teacherUserId);
    }

    I tried all kinds of scope definition, on annotation, on package/class patterns on extending super class pattrens etc, but nothing works. It seems, that interfaces are ignored at all by the ocelot agent. I would like to see, how long f.e. this find-method takes to complete.
    Thanks!

    Marius Oehler
    @mariusoe

    Hi @albersma69_twitter
    a scope for targetting the repository interface could look like this:

    inspectit:
      instrumentation:
        scopes:
          's_spring_repository':
            interfaces:
              - name: 'your.package.UserAgreementsRepository'
            advanced:
              instrument-only-inherited-methods: true

    this will result in an instrumentation of all the methods defined by your interface.

    If you want to target all methods of the implementing class, you can dismiss the advanced property of the example
    Bernd Albersmann
    @albersma69_twitter
    image.png
    Hi Marius,
    thank you. It works "somewhat", but not really satisfactory. The result is one new span inside the trace, which is called
    "$proxy201.findbyteacherid". The method name is correct, the class name not. I'd like to see the interface name in clear words instead of an anonymous proxy201, which implements that interface.
    Where can I find more advice /example on the inspectIt instrumentation grammar other than the little info, which I can find on https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/instrumentation? Thank you!
    Bernd Albersmann
    @albersma69_twitter
    How would I instrument all Repository Interfaces in my app across all packages, whose interace name ends with "Repository"? I tried this, but it does not work:
          'scope_KDO_repository':
            interfaces:
              - name: 'Repository'
                matcher-mode: 'ENDS_WITH'
            advanced:
              instrument-only-inherited-methods: true
    Marius Oehler
    @mariusoe

    Hi @albersma69_twitter

    yes, your approach for instrumenting all repositories whose name ends with the specified term is correct and should work.
    Note that the instrumentation is done asynchronously after the class was loaded. This means that the instrumentation is just active after some delay, thus, it may be the case that some of the spans are missing during this time.

    Regarding the span naming - this is actually not a wrong class name. The class is in fact named $proxy201 because it is generated by Spring under the hood. Due to the fact that the class name of the instrumented method is used as span name by default, your span is getting this name.
    Naming the span according to the interface is not trivial. Nevertheless it is possible. You can have a look on the following configuration where Spring repositories have been traced as well and the resulting spans named according to the defined interface: https://github.com/inspectIT/inspectit-ocelot-configurations/blob/master/extensions/spring-data-repository/spring-data-repository-profiling.yml
    In this case, the name is extracted from the proxy's arguments via Reflection.

    You can find more information on the instrumentation under the "Scopes" and "Rules & Actions" section in the documentation:

    Zhang Zhao
    @zzhao2010

    Hi,
    I ran the demo application provided in the inspectIT/Ocelot documentation (Prometheus and Jaeger Scenario) on my laptop. However after I issue the docker-compose command docker-compose -f docker-compose-prometheus-jaeger.yml up, the demo application was not running properly.. I saw this error message in the log:
    Caused by: com.netflix.client.ClientException: Load balancer does not have available server for client: customers-service
    api-gateway | 2021-05-28 00:10:38.460 WARN 1 --- [nio-8080-exec-4] o.s.c.n.z.filters.post.SendErrorFilter : Error during filtering
    api-gateway | com.netflix.zuul.exception.ZuulException: Forwarding error

    Anyone can help me out here? Thanks!

    Patrick-Eichhorn
    @Patrick-Eichhorn
    Hi @zzhao2010 . One possibility could be, that the docker container "customer-service" is not available (e.g. due to low memory or the service starts slower than the others). You can check this with the docker command docker ps after running docker-compose.
    Zhang Zhao
    @zzhao2010
    @Patrick-Eichhorn How do I fix this issue? I allocated 6 CPUs, 8GB Mem, and 2 GB Swap resources for Docker, which I think is more than the requirement in the doc.
    2 replies
    @Patrick-Eichhorn, yes.. I don’t see any service containers available..
    ❯ docker ps
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    e752bff17c71 openapm/spring-petclinic-api-gateway:inspectit-oce-demo "sh -c ' ./dockerize…" About a minute ago Up About a minute 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9094->9094/tcp, :::9094->9094/tcp, 8081/tcp api-gateway
    972983bf2ce9 openapm/spring-petclinic-discovery-server:inspectit-oce-demo "sh -c ' ./dockerize…" About a minute ago Up About a minute 0.0.0.0:8761->8761/tcp, :::8761->8761/tcp, 0.0.0.0:9095->9095/tcp, :::9095->9095/tcp discovery-server
    159725ed24ff openapm/spring-petclinic-config-server:inspectit-oce-demo "sh -c ' java -XX:+U…" 2 minutes ago Up About a minute 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp, 0.0.0.0:9096->9096/tcp, :::9096->9096/tcp config-server
    dea34d47cfd3 prom/prometheus:v2.7.1 "/bin/prometheus --c…" 2 minutes ago Up 2 minutes 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
    51779eba3eda inspectit/grafana:servicegraph-v2.1 "/run.sh" 2 minutes ago Up 2 minutes 0.0.0.0:3001->3000/tcp, :::3001->3000/tcp grafana
    73f6e4d31837 inspectit/inspectit-ocelot-agent:latest "sh /entrypoint.sh" 2 minutes ago Up 2 minutes agent
    18ab7aad2262 inspectit/spring-petclinic-load:1.0 "/artillery/docker-e…" 2 minutes ago Up 2 minutes load-generator
    22d2f3f7ded5 inspectit/inspectit-ocelot-configurationserver:latest "sh /entrypoint.sh" 2 minutes ago Up 2 minutes 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp ocelot-config-server
    ddd150c26958 jaegertracing/all-in-one:1.6 "/go/bin/standalone-…" 2 minutes ago Up 2 minutes 5775/udp, 5778/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp, :::16686->16686/tcp jaeger
    Alexander Krause
    @Alexander-Krause

    Hello,

    I use Ocelot to monitor the Spring Petclinic. My instrumentation file looks like this:

    inspectit:
    
      tags:
        extra:
          landscape_token: '36efef0d-30a4-48b9-a20b-a1735c2fdc53'
          token_secret: 'ilZsUSzJTJQPCmyk'
          application_name: 'petclinic-demo'
          application_instance_id: 0
          application_language: 'java'
    
      tracing:
        add-common-tags: ALWAYS
    
      metrics:
        enabled: false
    
      instrumentation:
        scopes:
          s_allClasses:
            type: 
              name: org.springframework
              matcher-mode: STARTS_WITH_IGNORE_CASE
    
        rules:
          r_all_methods_no_input_return:
            scopes:
              s_allClasses: true
            tracing:
              start-span: true
              attributes:
                java.fqn: method_fqn_attr
            entry:
              method_fqn_attr:
                action: a_get_method_fqn
    
        actions:
          a_get_method_fqn:
            input:
              _methodName: String
              _class: Class
            value: "new StringBuilder(_class.getName()).append('.').append(_methodName).toString()"
    
      logging:
        debug: true
        trace: false
    
      exporters:
        tracing:
          open-census-agent:
            address: "oc-collector:55678"
            service-name: "OC Agent Local"
            use-insecure: true

    Now, when using a scope broader than org.springframework.samples, i.e., org.springframework or org my analysis application does not receive data at all.

    Can you tell my why?

    Best regards,
    Alex

    Alexander Krause
    @Alexander-Krause

    And after a few minutes the petclinic cannot be reached at all and the log states an exception:

    Exception in thread "http-nio-8080-exec-2" java.lang.ClassCircularityError: rocks/inspectit/ocelot/core/instrumentation/context/InspectitContextImpl$1
        at rocks.inspectit.ocelot.core.instrumentation.context.ContextManager.enterNewContext(ContextManager.java:65)
        at rocks.inspectit.ocelot.core.instrumentation.hook.MethodHook.onEnter(MethodHook.java:80)
        at org.springframework.boot.loader.jar.Handler.parseURL(Handler.java:183)
        at java.base/java.net.URL.<init>(URL.java:674)
        at java.base/java.net.URL.<init>(URL.java:541)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader.checkResource(URLClassPath.java:848)
        at java.base/jdk.internal.loader.URLClassPath$JarLoader.getResource(URLClassPath.java:931)
        at java.base/jdk.internal.loader.URLClassPath.getResource(URLClassPath.java:314)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:455)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at rocks.inspectit.ocelot.core.instrumentation.context.ContextManager.enterNewContext(ContextManager.java:65)
        at rocks.inspectit.ocelot.core.instrumentation.hook.MethodHook.onEnter(MethodHook.java:80)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:123)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:204)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.bestEffortLoadClass(PackagingDataCalculator.java:228)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.computeBySTEP(PackagingDataCalculator.java:135)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(PackagingDataCalculator.java:100)
        at ch.qos.logback.classic.spi.PackagingDataCalculator.calculate(PackagingDataCalculator.java:58)
        at ch.qos.logback.classic.spi.ThrowableProxy.calculatePackagingData(ThrowableProxy.java:142)
        at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:122)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.log(Logger.java:765)
        at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
        at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
        at java.logging/java.util.logging.Logger.log(Logger.java:979)
        at java.logging/java.util.logging.Logger.doLog(Logger.java:1006)
        at java.logging/java.util.logging.Logger.logp(Logger.java:1283)
        at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:175)
        at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:141)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1725)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

    Again, this seems to be only the case when using a "broad" instrumentation scope. Its all running in Docker on MacOs with enough resources.

    Best regards,
    Alex

    1 reply
    kaushalshriyan
    @kaushalshriyan
    Hi All,
    Márton Szücs
    @martonsz
    Hi!
    How can I disable metrics for http_in_count? I get thousands of individual metrics for this because our API uses the path as a parameter.
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg1,...}
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg2,...}
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/arg3,...}
    ... thousands!
    http_in_count{environment="production",http_method="DELETE",http_path="/foo.bar/argN,...}
    1 reply
    I am using kubernetes configuration object
    # exampel for JMX
    ...
      metrics:
        jmx:
          enabled: true
          lower-case-metric-name: true
          object-names:
            # allow-listed (true) and deny-listed (false) object names
            '[Catalina:type=Connector,port=8080]': true
            '[Catalina:type=GlobalRequestProcessor,name="http-nio-8080"]': true
            '[Catalina:type=Manager,host=localhost,*]': true
    ...
    Matthias Huber
    @matthias-huber

    Hi, I am using the inspectIT Ocelot Standard configuration + a view to capture min and max values for HTTP In and out calls
    `inspectit:
    metrics:
    definitions:

      '[http/in/responsetime]':
        unit: ms
        views:
          '[http/in/responsetime]':
            aggregation: QUANTILES
            quantiles: [0, 1]
            tags: 
              'http_path': true
              'http_status': true
              'http_method': true
              'error': true
    
      '[http/out/responsetime]':
        unit: ms
        views:
          '[http/out/responsetime]':
            aggregation: QUANTILES
            quantiles: [0, 1]
            tags:
              'http_host': true
              'http_path': true
              'http_status': true
              'http_method': true
              'error': true

    `
    I get metrics but the average (calculated by sum/ count) is lower as the min value. Min and Max values seem to be reasonable (have to re-check by looking at Tracing Data).

    4 replies
    inspectit_avg_min_max.png
    Is my configuration of the views incorrect? Or do you have any idea what causes the difference, e.g. wrong PromQL query like mixing of different aggregation windows?
    Alexander Krause
    @Alexander-Krause

    Hello, the OpenTelemetry collector includes an OpenCensus receiver that seems to use this protobuf version for traces.

    I tried to send OC traces from Ocelot to the OpenTelemetry collector, but no data was received. I know that the OpenTelemetry support is still in development in Ocelot, but I wanted to try the backwards compatability of OC for the moment. Currently, I think that Ocelot uses a different protobuf version than the receiver, but couldn't figure out the version number.

    Did you successfully send traces from Ocelot to the OpenTelemetry collector in testing? Or is the protobuf version really in issue here?

    1 reply
    Matthias Huber
    @matthias-huber
    Hello guys,
    is it true that if I disable tracing with "-Dinspectit.tracing.enabled=false" the agent will not gather any HTTP Statistics (metrics)? Furthermore, is it true that if I set sampling_probalitity < 1.0 that also the HTTP Statistics are not covering all 100% of requests? In other words: Are HTTP Statistics only gathered if a request is actually traced?
    8 replies
    Matthias Huber
    @matthias-huber

    Hello guys,
    I have another situation that I don't know if my assumptions are true and if they are true I don't know how to deal with it. So what I try to solve is to extend the outgoing http metrics with another label that I need to distinguish calls to the same backend. Therefor I extend the available metric definition and set the value via the rule 'r_http_client_record_metric' as data-tag. I extract the tag somewhere during request processing (use up+down progagation before the backend call is initiated. In total this works. But I have an edge case after the initial start of the application where the label is empty for the first call (maybe more if they are executed fast after another -> have not tried that).

    I had a look at the instrumentation statistics from the agent before and after the request. The number of instrumented classes increases, probably as the classes involved in sending the request to the backend and also the class of the object the additional tag is extracted from is loaded. So I wonder if the label is not set because of the aynchronous instrumentation process. Meaning until the class hasn't change, the label will be empty. Can you confirm this behaviour? And if so is there any solution to it?

    I already thought of extracting the value deeper in the call stack so at best at the same class/method the other http statistics labels like http_path are extracted. Because they are available for the first call and the instrumentation points are already placed before the first request as the class has already been loaded?