Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Austin Parker
    @austinlparker
    @neith00 yes, this would be a good use case for tracing.
    neith00
    @neith00
    @austinlparker I was thinbking using prometheus as backend.
    Austin Parker
    @austinlparker
    ehh... probably? you'd have really high cardinality because you'd need a separate bucket per message.
    neith00
    @neith00
    well I don't think creating a bucket per message is worth unless I delete the bucket right after get rid of the message though
    In this case I should push the metrics to prometheus rather beeing scraped
    13217 Amasia
    @13217amasia
    Where can I find information on OT overhead/throughput loss over different sampling ratios? I understand this might be difficult question to answer but maybe there are some existing benchmarks...
    I also would like to learn if it's possible to control sampling ratio, when an application is just instrumented without any intrusive code changes? E.g. possibly by injecting a sampling ratio parameter to instrumentation parameters?
    Thank you!
    Naseem
    @naseemkullah
    This message was deleted
    2 replies
    Naseem
    @naseemkullah
    @13217amasia regarding sampling rate control: if using opentelemetry collector, sampling could be configured there (see https://github.com/open-telemetry/opentelemetry-collector/tree/master/processor#sampling-processor). If not it would be nice that a sampling rate be able to be passed as env var, I have a PR open for this in the js implementation: open-telemetry/opentelemetry-js#1069
    Ayushi Singh
    @ayusheesingh
    I'm trying to use the azuremonitorexporter but am getting an "invalid instrumentation key" error even though it matches the one I see for my Application Insights resource on Azure Portal - any recommendations on how to fix?
    Nicolas MacBeth
    @Nicolas-MacBeth
    Hey, not sure if this is where should ask this but here goes: It's my understanding that the Prometheus receiver for the OTel collector should just work with any Prometheus exporter, but I'm having issues with https://github.com/prometheus/mysqld_exporter . I get this message logged in the collector: {"level":"info","ts":1591373779.4030387,"caller":"internal/metrics_adjuster.go:357","msg":"Adjust - skipping unexpected point","type":"UNSPECIFIED"}, followed by (i think this is just Stackdriver exporter not happy with the prior error^) {"level":"info","ts":1591373782.0302048,"caller":"scrape/scrape.go:930","msg":"","level":"warn","scrape_pool":"mysql_exporter_otel","target":"http://localhost:9104/metrics","msg":"append failed","err":"[rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.; rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.; rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.; rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.; rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.]"}
    4 replies
    Does the mysql exporter need to be modified to format the metrics according to the collector, or is there something I'm missing in the collector config?
    dcwangmit01
    @dcwangmit01_twitter

    Hi. I've got comments/questions after a few days of playing with OTel.

    1) Why is UpDownCounter instrument not called Gauge? The six instrument types all have very similar naming components (Counter, UpDownCounter, ValueRecorder, SumObserver, UpDownSumObserver, ValueObserver). I'm sure I'll get used to this over time, but as a newb the naming pattern is hard to spot and requires a lot of spec checking while coding.

    2) The UpDownCounter instrument, which is equivalent to Gauge, is updated via an Add() function. How do I update this counter when I already have the final value? Right now it appears to me that I need to track the prior value, do the math to find the delta, and then call the Add() function with the delta as the value. I would rather just update the metric with the current value. Is there a way to directly set the value for an UpDownCounter instrument?

    3) It took me a while to realize that ValueRecorder and ValueObserver instruments were summary metrics that had multiple values (MinMaxSumCount), and thus would be dropped by the OTel exporter for Prometheus. To me, "Value" suggests a single scalar value. I would have expected the exporter for Prometheus to export the LastValue and just work for Value* instruments. This burned at least a day of debugging for me.

    Thanks all, for this amazing project. I've read some of the spec discussions and the decisions made have all been well considered.

    John Watson
    @jkwatson
    @dcwangmit01_twitter I'm not sure I get why you say that UpDownCounter is the same as a Gauge. From what I know of gauges, they generally only aggregate as the LastValue, not as a sum.
    23 replies
    Mouliksoni
    @Mouliksoni
    Hi, I have a quick question about exporter config (agent side). I'm looking to specify certain attributes to the exporter, I see that there is an way to specify attributes when I create a new instance of the exporter (in my case new new CollectorExporter for nodejs). I was looking for a way to specify the attributes via a config file instead, like how it's possible to do it on the collector side?
    Austin Parker
    @austinlparker
    @Mouliksoni not at this time, although I think there's some interest in such a thing
    3 replies
    Naseem
    @naseemkullah

    What are pros/cons of having a trace begin at a load balancer rather than at the first hop server-side?

    Presumably the trace can begin before that, at the client device, what are pros/cons of having the root span there?

    Would be nice to have a chart listing these three locations where a trace can begin along with pros/cons
    4 replies
    Erik LaBianca
    @easel
    Is there a convention in opentelemtry for exposing the trace id / span id in http responses? I’d like to be able to hit an api endpoint, grab the trace id from the header and drop that into jaeger or whatever and see the trace.
    Austin Parker
    @austinlparker
    i'll give a pretty strong "it depends" here, but if you send trace context to an endpoint then you'd get the trace ID in the headers on the response
    you'd have to look at the instrumentation for the HTTP/gRPC framework - if it injects the trace context into the response even on the root span and you don't have anything stripping those headers, I don't see why not
    John Watson
    @jkwatson
    I think I'd be pretty surprised to see instrumentation inject headers into a response
    Can you show some instrumentation that does that, @austinlparker ?
    Anthony Mirabella
    @Aneurysm9
    the Go net/http instrumentation does, though I'm not positive it's a great idea
    trace context is probably fine, baggage maybe not
    John Watson
    @jkwatson
    what span id does it inject?
    Anthony Mirabella
    @Aneurysm9
    that would be the span ID of the span created by the instrumentation wrapper
    John Watson
    @jkwatson
    is that going to be useful to the caller, I wonder?
    I don't see in that code where it's injecting headers into the response (I also don't really read go, so I might be missing it)
    Anthony Mirabella
    @Aneurysm9
    the first link was to a ResponseWriter wrapper that wraps the WriteHeader method to trap the status code and inject into the outbound Header
    the context it uses is created at the second link, where the instrumentation creates a span for an incoming request
    John Watson
    @jkwatson
    gotcha
    Austin Parker
    @austinlparker
    if there's an incoming span context than the instrumentation will inject it into the response, but its also one of those "well why would it matter" because at that point you've got the trace ID from the outgoing request
    John Watson
    @jkwatson
    I thought that the W3C spec had specifically removed this "backward propagation" case, but I don't know if it explicitly rejects it from being done.
    Austin Parker
    @austinlparker
    i'm also not sure, i'd need to do some research
    John Watson
    @jkwatson
    But, this is only what my probably faulty memory tells me
    Robert Wolujewicz
    @robertwol
    hello Community.
    How one add/include the collector community contrib receivers/exporters to the collector code/image for personal use?
    2 replies
    Lisa Crispin
    @lisacrispin
    The teams in our R&D org are being pushed to get going with OpenTelemetry and observability. A few teams have done several really small experiments with OpenTel, some using Honeycomb but most using ElasticSearch. Our code is C# and JS. (And low code, as the product is a low code platform, but let's not go there yet). Do any of you have recommended links for devs who are total newbies to get going with OpenTel in .Net? I have a list of my own but I'm not a coder and not an expert on this, I'm just trying to support people with good information.
    2 replies
    Amol
    @Amolstar_twitter
    Hi, We have a requirement to store the metrics in Oracle database. Please let me know if we have an exporter for storing this? Any suggestions on how to go about this? Thanks
    Amol
    @Amolstar_twitter
    Hi, Any suggestions on the above query. Thanks.
    John Watson
    @jkwatson
    I don't believe that a metric exporter that writes to a SQL database has been implemented by anyone.
    1 reply
    Chris Young
    @worldofchris
    Hi, I am a complete newbie in the world of open-telemetry. Using the Python libs I have set it up to instrument a video processing workflow which is made up of a number of Google Cloud Functions which communicate with each other via http and Pub/Sub. I love the potential of Open Telemetry and am excited by the results I'm getting with what I've put in place so far. Where in the Open Telemetry community is the best place to share this experience and get feedback on whether I am going in the right direction? Aware that I could be misunderstanding some things / cutting against the grain. Thanks!
    Chris Young
    @worldofchris
    @zeus92bgr is that 'Nf' in response to my question above? Thanks.
    Oh it's been deleted now.
    Andrey Afoninsky
    @afoninsky
    hello all
    I'm trying to deliver traces into two system (jaeger / grafana tempo) in order to play with them, so I use opentelemetry-collector as a receiver
    when I send traces directly to collector - I see them in logs but downstream services don't have them (if I send directly to jaeger everything is ok)
    could you point the way how can I debug it and see where I failed? config and commands I use: https://gist.github.com/afoninsky/7496c4bb89cc461aa33ffeb64318a634
    thank you
    Lucas Harms
    @lmickh
    Are there any examples of using the k8sprocessor on the collector?
    Melissa Fritcher
    @melissafritcher
    Not sure if this q should go here or in the python channel. I'm instrumenting a uwsgi/Flask app using a ConsoleSpanExporter with a BatchExportSpanProcessor. Just trying to get a hello world working at the moment. when I run the flask app directly using python3 server.py --log=DEBUG the spans emit just fine. but when I'm running using uwsgi I can't get the spans to emit to console. I've tried setting logging level in logging.basicConfiguration but no luck. Any ideas?
    1 reply
    Leonardo Nerone
    @LeoNero
    Hi everyone. Sorry if this is not the right place to ask. I want to build my own opentelemetry backend (for example, create a mini zipkin clone) so that I can build my own tracing visualization. I am really new to OpenTelemetry, and right now, I am kinda confused on how to do that. Do I need to create my own exporter? Do I have to setup my own collector? All the information I found is kinda confusing, so any guidance would be really helpful :) thank you
    5 replies
    Fabien LEBAILLIF-DELAMARE
    @fabienLD_gitlab
    Hi, I am trying to export hostmetrics to a running Prometheus instance using the collector, I am not sure how to configure the exporters: section in the yaml, my Prometheus can fetch from the collector:8888/metrics endpoint but the only thing present related to host metrics for instance is otelcol_scraper_scraped_metric_points{receiver="hostmetrics",scraper="paging",service_instance_id="bfb07d37-28af-4fa4-90db-94694478e5d1"} 9, it must be something silly I am not doing right ?
    1 reply
    Naseem
    @naseemkullah
    Is it typically better to sample at the application or remotely (e.g. otel-collector) ? What are some of the trade offs?
    1 reply
    madvikinggod
    @madvikinggod:matrix.org
    [m]
    I would only sample if you have a need. Then you can make the trade off. Until then you are optimizing unnecessarily.
    Naseem
    @naseemkullah
    Great read on head based vs tail based sampling which answers the question I posed above: https://blog.kroehling.de/the-role-of-sampling-in-distributed-tracing