open-telemetry/getting-started

Austin Parker

@austinlparker

oh, weird. I guess it's not in sync yet

Naseem

@naseemkullah

we unfortunately do not control the opentelemetry namespace on docker hub

Ah too bad. Regardless, what about dropping the preceding opentelemetry- in the image names? And in fact just have 1 image for the collector with various tags. e.g.: otel/collector and tags such as latest, stable, contrib, dev and such? Pardon the nitpicking :)

Naseem

@naseemkullah

Also, I'd like to propose a helm chart for the collector, any chance we could create an opentelemetry/helm-charts repo?

Austin Parker

@austinlparker

I know there's a Kubernetes operator for the collector already, but I'm not sure if there's a helm chart. These are good questions! I'd open an issue for them on the collector repo.

Naseem

@naseemkullah

WIll do, thanks!

Steve Flanders

@flands

No helm chart yet — PRs welcomed!

Naseem

@naseemkullah

sounds good! I've created an issue open-telemetry/opentelemetry-collector#752 asking for a helm-charts repo.

Naseem

@naseemkullah

I'm having issues chaining spans together, nginx ingress is using jaeger client to instrument, and can chain to another jaeger instrumented service, but it does not seem to chain with otel instrumented services that also emit spans. Any ideas?

2 replies

neith00

@neith00

I'm actively trying to better monitor a distributed system which has a message broker between its component. My idea, is to provide a tool to component developers to track the messages that goes through the system. For instance, sometimes during development we (hardly) notice that some messages are lost/dropped/whatever during the process. Could OpenTelemetry help me to build a tool which would give us the status/location of each messages going through our system?

Austin Parker

@austinlparker

@neith00 yes, this would be a good use case for tracing.

neith00

@neith00

@austinlparker I was thinbking using prometheus as backend.

Austin Parker

@austinlparker

ehh... probably? you'd have really high cardinality because you'd need a separate bucket per message.

neith00

@neith00

well I don't think creating a bucket per message is worth unless I delete the bucket right after get rid of the message though

In this case I should push the metrics to prometheus rather beeing scraped

13217 Amasia

@13217amasia

Where can I find information on OT overhead/throughput loss over different sampling ratios? I understand this might be difficult question to answer but maybe there are some existing benchmarks...
I also would like to learn if it's possible to control sampling ratio, when an application is just instrumented without any intrusive code changes? E.g. possibly by injecting a sampling ratio parameter to instrumentation parameters?
Thank you!

Naseem

@naseemkullah

This message was deleted

2 replies

Naseem

@naseemkullah

@13217amasia regarding sampling rate control: if using opentelemetry collector, sampling could be configured there (see https://github.com/open-telemetry/opentelemetry-collector/tree/master/processor#sampling-processor). If not it would be nice that a sampling rate be able to be passed as env var, I have a PR open for this in the js implementation: open-telemetry/opentelemetry-js#1069

Ayushi Singh

@ayusheesingh

I'm trying to use the azuremonitorexporter but am getting an "invalid instrumentation key" error even though it matches the one I see for my Application Insights resource on Azure Portal - any recommendations on how to fix?

Nicolas MacBeth

@Nicolas-MacBeth

Hey, not sure if this is where should ask this but here goes: It's my understanding that the Prometheus receiver for the OTel collector should just work with any Prometheus exporter, but I'm having issues with https://github.com/prometheus/mysqld_exporter . I get this message logged in the collector:

{"level":"info","ts":1591373779.4030387,"caller":"internal/metrics_adjuster.go:357","msg":"Adjust - skipping unexpected point","type":"UNSPECIFIED"}

, followed by (i think this is just Stackdriver exporter not happy with the prior error^)

{"level":"info","ts":1591373782.0302048,"caller":"scrape/scrape.go:930","msg":"","level":"warn","scrape_pool":"mysql_exporter_otel","target":"http://localhost:9104/metrics","msg":"append failed","err":"[rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.; rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.; rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.; rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.; rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted.]"}

4 replies

Does the mysql exporter need to be modified to format the metrics according to the collector, or is there something I'm missing in the collector config?

dcwangmit01

@dcwangmit01_twitter

Hi. I've got comments/questions after a few days of playing with OTel.

1) Why is UpDownCounter instrument not called Gauge? The six instrument types all have very similar naming components (Counter, UpDownCounter, ValueRecorder, SumObserver, UpDownSumObserver, ValueObserver). I'm sure I'll get used to this over time, but as a newb the naming pattern is hard to spot and requires a lot of spec checking while coding.

2) The UpDownCounter instrument, which is equivalent to Gauge, is updated via an Add() function. How do I update this counter when I already have the final value? Right now it appears to me that I need to track the prior value, do the math to find the delta, and then call the Add() function with the delta as the value. I would rather just update the metric with the current value. Is there a way to directly set the value for an UpDownCounter instrument?

3) It took me a while to realize that ValueRecorder and ValueObserver instruments were summary metrics that had multiple values (MinMaxSumCount), and thus would be dropped by the OTel exporter for Prometheus. To me, "Value" suggests a single scalar value. I would have expected the exporter for Prometheus to export the LastValue and just work for Value* instruments. This burned at least a day of debugging for me.

Thanks all, for this amazing project. I've read some of the spec discussions and the decisions made have all been well considered.

John Watson

@jkwatson

@dcwangmit01_twitter I'm not sure I get why you say that UpDownCounter is the same as a Gauge. From what I know of gauges, they generally only aggregate as the LastValue, not as a sum.

23 replies

Mouliksoni

@Mouliksoni

Hi, I have a quick question about exporter config (agent side). I'm looking to specify certain attributes to the exporter, I see that there is an way to specify attributes when I create a new instance of the exporter (in my case new new CollectorExporter for nodejs). I was looking for a way to specify the attributes via a config file instead, like how it's possible to do it on the collector side?

Austin Parker

@austinlparker

@Mouliksoni not at this time, although I think there's some interest in such a thing

3 replies

Naseem

@naseemkullah

What are pros/cons of having a trace begin at a load balancer rather than at the first hop server-side?

Presumably the trace can begin before that, at the client device, what are pros/cons of having the root span there?

Would be nice to have a chart listing these three locations where a trace can begin along with pros/cons

4 replies

Erik LaBianca

@easel

Is there a convention in opentelemtry for exposing the trace id / span id in http responses? I’d like to be able to hit an api endpoint, grab the trace id from the header and drop that into jaeger or whatever and see the trace.

Austin Parker

@austinlparker

i'll give a pretty strong "it depends" here, but if you send trace context to an endpoint then you'd get the trace ID in the headers on the response

you'd have to look at the instrumentation for the HTTP/gRPC framework - if it injects the trace context into the response even on the root span and you don't have anything stripping those headers, I don't see why not

John Watson

@jkwatson

I think I'd be pretty surprised to see instrumentation inject headers into a response

Can you show some instrumentation that does that, @austinlparker ?

Anthony Mirabella

@Aneurysm9

the Go net/http instrumentation does, though I'm not positive it's a great idea

trace context is probably fine, baggage maybe not

John Watson

@jkwatson

what span id does it inject?

Anthony Mirabella

@Aneurysm9

that would be the span ID of the span created by the instrumentation wrapper

John Watson

@jkwatson

is that going to be useful to the caller, I wonder?

I don't see in that code where it's injecting headers into the response (I also don't really read go, so I might be missing it)

Anthony Mirabella

@Aneurysm9

the first link was to a ResponseWriter wrapper that wraps the WriteHeader method to trap the status code and inject into the outbound Header

the context it uses is created at the second link, where the instrumentation creates a span for an incoming request

John Watson

@jkwatson

gotcha

Austin Parker

@austinlparker

if there's an incoming span context than the instrumentation will inject it into the response, but its also one of those "well why would it matter" because at that point you've got the trace ID from the outgoing request

John Watson

@jkwatson

I thought that the W3C spec had specifically removed this "backward propagation" case, but I don't know if it explicitly rejects it from being done.

Austin Parker

@austinlparker

i'm also not sure, i'd need to do some research

John Watson

@jkwatson

But, this is only what my probably faulty memory tells me

Robert Wolujewicz

@robertwol

hello Community.
How one add/include the collector community contrib receivers/exporters to the collector code/image for personal use?

2 replies

Lisa Crispin

@lisacrispin

The teams in our R&D org are being pushed to get going with OpenTelemetry and observability. A few teams have done several really small experiments with OpenTel, some using Honeycomb but most using ElasticSearch. Our code is C# and JS. (And low code, as the product is a low code platform, but let's not go there yet). Do any of you have recommended links for devs who are total newbies to get going with OpenTel in .Net? I have a list of my own but I'm not a coder and not an expert on this, I'm just trying to support people with good information.

2 replies

Amol

@Amolstar_twitter

Hi, We have a requirement to store the metrics in Oracle database. Please let me know if we have an exporter for storing this? Any suggestions on how to go about this? Thanks

Amol

@Amolstar_twitter

Hi, Any suggestions on the above query. Thanks.

John Watson

@jkwatson

I don't believe that a metric exporter that writes to a SQL database has been implemented by anyone.

1 reply

Chris Young

@worldofchris

Hi, I am a complete newbie in the world of open-telemetry. Using the Python libs I have set it up to instrument a video processing workflow which is made up of a number of Google Cloud Functions which communicate with each other via http and Pub/Sub. I love the potential of Open Telemetry and am excited by the results I'm getting with what I've put in place so far. Where in the Open Telemetry community is the best place to share this experience and get feedback on whether I am going in the right direction? Aware that I could be misunderstanding some things / cutting against the grain. Thanks!

Where communities thrive

People

Repo info

Activity