by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 31 2019 19:53
    wafflebot[bot] unlabeled #1280
  • Jan 31 2019 19:53
    vprithvi closed #1280
  • Jan 31 2019 19:53

    vprithvi on master

    Implement FindTraceIDs function… (compare)

  • Jan 31 2019 19:05
    codecov[bot] commented #1280
  • Jan 31 2019 19:02
    codecov[bot] commented #1280
  • Jan 31 2019 18:51
    codecov[bot] commented #1280
  • Jan 31 2019 18:51
    vprithvi synchronize #1280
  • Jan 31 2019 16:18
    pavolloffay edited #1309
  • Jan 31 2019 14:58
    wafflebot[bot] labeled #1242
  • Jan 31 2019 14:58
    wafflebot[bot] assigned #1242
  • Jan 31 2019 14:58
    wafflebot[bot] labeled #1309
  • Jan 31 2019 14:58
    wafflebot[bot] assigned #1309
  • Jan 31 2019 14:58
    pavolloffay review_requested #1309
  • Jan 31 2019 14:58
    pavolloffay review_requested #1309
  • Jan 31 2019 14:57
    pavolloffay review_requested #1309
  • Jan 31 2019 14:57
    pavolloffay review_requested #1309
  • Jan 31 2019 14:57
    pavolloffay review_requested #1309
  • Jan 31 2019 14:57
    pavolloffay review_requested #1309
  • Jan 31 2019 14:57
    pavolloffay opened #1309
  • Jan 31 2019 08:21
    vlamug commented #1280
varun111111
@varun111111
Hello....Can i run Jaeger in kubernetes in Openshift cloud?
@@@@
@narendra0356
We have run the performance test with ES backend collector & query and Cassandra backend collector & query, For complete the PT with Cassandra endpoint collector &query has taken more time compare ES endpoint collector & query. Based PT results, ES backend is give the best performance? How to increase the query performance while using the Cassandra endpoint?
3 replies
Veera P
@veera83372
is it possible to export application logs along with span in jaeger, if yes then how?
oigumbor-tc
@oigumbor-tc
I'v got an app that uses the jaeger-client, with jaeger-agent and jaeger-collector all as standalone docker containers. If i use thrift msg port btw client and agent, can i use grpc btw agent and collect? In order words, is there a requirement that only one protocol should be used all the way through from client to collector?
Nick Lanham
@nicklan
Is there a way I can limit the number of traces jaeger-query ties to fetch from elasticsearch? I se es.max-num-spans to 10000, but i ran a query with "Limit Results" set to 100, which fetched 100 traces totaling 117070 spans, so I guess that's a per trace limit or something?
7 replies
or if that's not possible, what's the suggested way to prevent OOMing on jaeger-query when a "big" query comes in
Pandiaraj2911
@Pandiaraj2911
what is the calculation between queue size and worker count for 2 lakhs spans job?
I had set to queue size 50K and worker count 100 , 30% of spans are dropped.
Is there any calculation with respect to span count?
Romil Punetha
@romilpunetha

Is there a way to show databases as a dedicated fan-out in jaeger, just like in datadog?

Screenshot 2020-05-22 at 7.14.38 PM.png

27 replies
Screenshot 2020-05-22 at 7.14.14 PM.png
Screenshot 2020-05-22 at 7.14.27 PM.png
Nick Lanham
@nicklan
i'm trying to debug a failure to connect to elasticsearch. i've specified --log-level=debug but I get no debug output. how do i get debug info?
1 reply
mkt32
@mkt32
Help Please .. RFC 4122 says that a UUID is 128 bits long and can guarantee
uniqueness through space and time. Does tracing_ID 64 or 128 guarantee the same?
jalpeshshelar
@jalpeshshelar
Can we get end user's Public IP in span details?? I have Nginx before Go layer
1 reply
Romil Punetha
@romilpunetha

Any ideas why findTraces would be taking 200 seconds?

Screenshot 2020-05-25 at 3.59.01 AM.png

Romil Punetha
@romilpunetha

Any idea why memory for collector is varying so much across pods?

Screenshot 2020-05-25 at 3.49.41 PM.png

2 replies
Carl Henrik Lunde
@chlunde
What have you done about PII in jaeger/traces? Anything around multi-tenancy/access control?
Pavol Loffay
@pavolloffay
Not much, there are a couple of blog posts around multitenancy
Jonah Kowall
@jkowall
Is there any form of a standard to export metrics/aggregates from Jaeger to another metric system? For example Prometheus for managing/tracking SLAs with trace data versus doing that at other infra layers or building custom metrics in the code?
@chlunde You can do this in your backend, but you'd need to handle it on your own. We do this in our platform for example. We have tenants, and sub-tenants and the data is permissioned accordingly on the backend where Jaeger can only see the data it's permissioned for...
Romil Punetha
@romilpunetha

when im directly querying elasticsearch with some queries, thats pretty fast. However, the

Any ideas why findTraces would be taking 200 seconds?

Screenshot 2020-05-25 at 3.59.01 AM.png

This is still bugging me. The /api/traces call takes over a min, while the /api/traces/{traceID} takes almost 15 seconds, out of which multiread is just 0.14ms. Not sure where the rest of the time is going.

Carl Henrik Lunde
@chlunde
@jkowall @pavolloffay yeah, so splitting two companies is pretty easy with multiple deployments, for example with two jaeger agents daemonsets with different hostPort. But it gets more difficult/interesting when there are multiple cooperating companies, and maybe you want to allow looking at traces (at least span start/stop time and service name) across tenants. Also, I think as a service owner it would be nice to be able to see all parent spans for your trace, but you may not need/have access to all siblings. Another approach is to avoid having any PII in jaeger at all, but there might be some parts info in URLs, client IP, and more anyway so it is difficult to be sure that there's not PII left anywhere... I think this (and auto-masking obvious PII) could be implemented as a "storage filter plugin" in the query engine... but unless many people want this, or they want more or less the same thing, I don't think it's a good idea to go there :)
Deniz Gurkaynak
@dgurkaynak
Hey guys, I'm not sure if this is the right medium, but I've built a trace analyzer tool for Jaeger and Zipkin, and I'm really excited about finally releasing it after working on it for over a year now. The main motivation behind it is to provide an alternative way to inspect & visualize traces, besides Jaeger and Zipkin's built-in UI. It's fully open-source, and I hope you'll find it useful. Any feedback is very welcome. https://github.com/dgurkaynak/stalk-studio
6 replies
Romil Punetha
@romilpunetha
Screenshot 2020-05-26 at 12.39.17 PM.png
Screenshot 2020-05-26 at 12.39.37 PM.png
Can anyone help me with the optimal config for collector? I observe high memory usage, low cpu usage, frequent container restarts, and span drops. Backend is ES with st1 EBS volume
11 replies
Saeed Ataee
@Saeed-Ataee
I read jaegertracing/jaeger#2188. Is it just updating strategy from file? Can I update strategy by code?? (programmatically)
mkt32
@mkt32
RFC 4122 says that a UUID is 128 bits long and can guarantee uniqueness through space and time.
Does tracing_ID 64 or 128 guarantee the same?
My concern would be. What is the probability of duplication of traceid with clustered jaeger?
Example: Could have physical machines / containers
8 replies
Romil Punetha
@romilpunetha
can someone explain the difference between --collector.num-workers and --es.bulk.workers flags for jaeger collector? --collector.num-workers says that its the number of workers that pick spans from the queue, and do what with them? If they're supposed to send them to elasticsearch, then whats --es.bulk.workers?
2 replies
Devesh Bajpai
@deveshbajpai19
Hello everyone, I am investigating on having a tracing solution for a piece of an application which has LMAX Disruptor (https://github.com/LMAX-Exchange/disruptor). Does anyone have a prior experience in adding OpenTracing hooks in it?
Romil Punetha
@romilpunetha

Even if the collector scales, the new pods take an enormous amount of time to catchup. Does not happen once you rollout restart the deployment.

Screenshot 2020-05-27 at 3.10.08 PM.png

7 replies
prashanthga
@prashanthga
when i configure mTLS for the jaeger components, the collector fails with unable to connect to initial hosts
1 reply
please help
Hashfyre
@Hashfyre
Hi folks, I started out in the jaeger/ tracing ecosystems a few weeks back, I wrote a sample golang app with jaeger traced APIs, and am trying to setup the jaeger stack both locally and k8s.
The local docker-compose stack has been failing to register traces. My app says the span has been created, but I don’t see anything in the jaeger-queryUI.
I have gone through the docker-compose in the repo and based mine off that.
Sharing the docker-compose in thread
126 replies
Naveed Yahya-Zadeh
@naveedyahyazadeh_gitlab

Good afternoon, I'm currently benchmarking the Jaeger client's performance impact on my system by running a comparison of the Jaeger client metrics when the Jaeger agent is on/off. The specific metric I am interested in is linked below. When my Jaeger agent is active, 100% of my spans are successfully reported (result = ok), as is expected. When the agent is off, however, i'm getting an almost 50/50 split of ok/error labels on the metric. I expected to see this metric yield 100% error, since the client has no agent to send its spans to. Any tips on how to debug this discrepancy, or information on the metric itself?

https://github.com/jaegertracing/jaeger-client-java/blob/af610ad36fa2fcf99a831e6ee5b12e982a29ba74/jaeger-core/src/main/java/io/jaegertracing/internal/metrics/Metrics.java#L157

1 reply
Romil Punetha
@romilpunetha

Anyone having a better configuration for jaeger collector? My queue isnt clearing fast enough.

options:
      collector:
        num-workers: "20000"
        queue-size: "500000"
      es:
        num-shards: 8
        tags-as-fields:
          all: false
        bulk:
          workers: "50"
          flush-interval: "200ms"
pods: 30
resources:
      requests:
        cpu: 200m
        memory: 3Gi
      limits:
        cpu: "500m"
        memory: 4Gi

Screenshot 2020-05-28 at 12.31.52 PM.png

9 replies
Ido Halevi
@idohalevi
Hi, what is the best approach to instrument my code in a serverless architecture?
Alexandre
@agaudreault
Hi, this question is about Adaptive Sampling. In our collector, we define the remote sampling configuration strategies and for some default endpoints (/health, /metrics, etc.), we set the probabilistic sampling rate of 0.0. From what I found when debugging, the client librairies will create a GuaranteedThroughputSampler whenever a list of operations is received. The default lowerbound value will always be of 0, but when the RateLimiterSampler is created, if the value is smaller than 1, the default value will be 1. This cause the /health, /metrics, etc. endpoints to be sampled at a rate of 1 TPS. Is there currently a way to disable the sampling for specific endpoints ? Or perhaps it is a bug that that the RateLimiterSampler cannot be instantiated with a value of 0 ?
We are using Jaeger collector 1.8 and the latest release of jaeger-client-java and jaeger-client-go.
craynic
@craynic
I found Jaeger tracer does not inject debug-id headers into carrier. Is it by design?
andrew bednarz
@andrew_bednarz_twitter
Hey Guys, I'm trying to get the operator up and running on openshift 3.11 k8s 1.11 and I'm getting an error like this: The CustomResourceDefinition "jaegers.jaegertracing.io" is invalid: spec.validation.openAPIV3Schema: Invalid value: apiextensions.JSONSchemaProps{ID:"", Schema:"", Ref:(string)(nil), Description:"", Type:"object", Format:"", Title:"", Default:(apiextensions.JSON)(nil) when trying to create the CRD (create -f jaegertracing.io_jaegers_crd.yaml) has anyone run into this? Thanks in advance for the help
Romil Punetha
@romilpunetha
Is there a way to configure elasticsearch index that is created by the collector using jaeger operator? I wanted to reduce the index refresh interval from a default of 1 sec to 30 seconds. However, when i manually go and edit the index template, it resets to 1 sec after some time.
3 replies
Ned Ellis
@ned-db

Is there a way to make the Jaeger Agent send out its collected traces using "jaeger.thrift (14268)" format? I am working on a system with the following constraints:

  • Trace producer is a NGINX reverse proxy that exports Jaeger traces with COMPACT jaeger.thrift protocol over UDP. I tried to configure it to output "jaeger.thrift" directly but the functionality seems broken and the software does not seem maintained: opentracing-contrib/nginx-opentracing#129 so I am stuck with using the UDP export for now.
  • Collector is an already running Jaeger Collector. For security and infrastructure reasons it can only accept "jaeger.thrift (14268)" at the moment

Ideally I'd like to use the Jaeger Agent in the middle as an intermediary, accepting COMPACT jaeger.thrift and outputting HTTP jaeger.thrift but it seems like there is not a way to do that currently, as the Collector can only output the gRPC flavor traces. Has anyone run into a similar situation? Thanks :)

9 replies
Paul DeMarco
@pauldemarco
Hi everyone. I've managed to get jaeger initializing nicely with the propagated headers, and i'm seeing jaeger logging output in Kiali. Is there a way to initialize a Zap logger using any of these existing resources? In other words, how best can I use a zap logger with this setup?
Paul DeMarco
@pauldemarco
Or is it generally a bad idea to mix general logging using Zap and tracing using Jaeger?
3 replies
Angel Lopez Bellmont
@AngelLopezBellmont
hi I am totally new in jaeger. My company just install it. I need to learn. Where (with which tutorial) should I start?
11 replies
image.png
image.png
image.png
Nathan
@phxnsharp
Hello! We are having an issue with a short-lived managed C++ process instrumented with Jaeger CSharp 0.3.6. Sometimes some of the last spans do not get reported to the agent running on localhost. We are calling ITracer.Dispose() before the process exits. Reading the code, it looks like it at least attempts to let any queued spans get sent before returning, so I'm a little baffled at the moment. Has anyone else experienced anything similar?
@@@@
@narendra0356
In jaeger-query, all application teams can access the UI dashboard, means each application team is able view the another application service names and their traces. In jaeger-Query UI, one application team can able to view their services name and traces only. Is there any way for segregating apps services in UI like metrics?
3 replies
MatrixGod-max
@MatrixGod-max

hello folks, I'm running spark-dependencies in a docker container, it seems that it can't run normally, exit after 5 seconds, my command is docker run -d --env STORAGE=elasticsearch --env ES_NODES=http://192.167.0.x: 9600 jaegertracing/spark-dependencies:latest

and the docker log is:
20/06/04 06:59:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/06/04 06:59:03 INFO ElasticsearchDependenciesJob: Running Dependencies job for 2020-06-04T00:00Z, reading from jaeger-span-2020-06-04 index, result storing to jaeger-dependencies-2020-06-04
20/06/04 06:59:04 INFO ElasticsearchDependenciesJob: Done, 0 dependency objects created