open-telemetry/community

THIS FORUM IS NO LONGER ACTIVE. Join us on CNCF Slack: https://cloud-native.slack.com/archives/CJFCJHG4Q.

Bruno Bonfils

@asyd

hello, anyone use flask intrusmentation by chance? I have few questions. How to add custom tags for example, and if it possible to add system metrics inside a trace

1 reply

Gutemberg Ribeiro

@galvesribeiro

I'm using JaegerExporter for .Net and all my telemetry is reporting as "OpenTelemetry Exporter" as you may see on this picture on jaeger UI.
What should be set on the telemetry reports or the exporter in order to have the proper service name there?
Thanks!

Just found it, thanks!

deyan-georgiev

@deyan-georgiev

Hello,
I was curious as per the standard, are labels meant to be per value or per metric?
As a concrete example, I want all HTTP requests sent from an application to be exposed as two metrics - requests count(as Counter) and latency(as ValueRecorder). The application is sending requests to a verity of 3rd party systems, I want the metric to contain a way to group the request based on DNS name, response code, path and others. Is labels the correct abstraction to allow such grouping or there is something else, better suited?

P.S If this is not the correct place to ask this question, could somebody point me to the correct one?

2 replies

Buchi Reddy Busi Reddy

@buchireddy

Hello, I'm curious about what the community thinks about using OTel libraries for this use case:
In microservices world, we need to propagate the Authorization header across different services, just like tracing headers are propagated. I think we can use OTel's Propagator implementation for this, when the services are using tracing with OTel. Is someone doing that already?

7 replies

gagansingh355

@gagansingh355

Hey, My question is regarding fluentbit. Does anyone know what's the format of logs in which fluentbit send data to the OTEL exporter?

1 reply

BuRnCycL

@burncycl

Hello OTEL community.If I want to exclude a span received from being exported, is there a means to do that using span processor. Example:

span/1:
        span:
          exclude:
            match_type: regexp
            span_names:
              - .*/health

1 reply

Matt

@pabrahamusa

I just started looking at open-telemetry for Java apps , Wondering whether there are any open source tool that do visualizations like LightStep?

Kenny Long

@kennethlong

Anybody have an OpenTelemetry overview slidedeck, something like Steve Flanders "How OpenTelemetry is eating the World" @flands . Thanks in advance!

4 replies

filip popa

@fpopa_gitlab

Hi everyone :)

I'm looking for an example of using open telemetry in a go grpc project, I think it should be doable to add it inside 2 receiving interceptors, unary and stream.

Unfortunately I've only found examples using open tracing

Jamie Snell

@jamiegs

What APMs are people using with OpenTelemetry? I tried out a few of them, Sumo Logic, Sentry.io , SignalFx, Light Step.. They seem to have varying levels of OpenTelemetry support.. some don’t support things like reading the environment out of the traces or that sentry doesn’t seem to work great for traces across multiple applications.
I’m looking for an APM that follows the OpenTelemetry specification well and has things like operation performance over time other sorts of trend analysis

Juraci Paixão Kröhling

@jpkrohling

@jamiegs jaeger won't give you that kind of analysis, but people have been using it with Kafka as storage and then creating custom consumers to extract the information they need. As you might indeed want an APM from a SaaS provider, you are missing as couple from your list: honeycomb, datadog, splunk, aws ... Perhaps a good source is to go over the list of otel-collector exporters and check which ones are related to commercial vendors? https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/master/exporter

Steve Flanders

@flands

@jamiegs Also note that OpenTelemetry is not GA yet and the spec is still changing. Once GA hits vendors should begin to fully support the spec and its features. If anything is missing on the Splunk/SignalFx side let me know and happy to assist.

2 replies

Ghost

@ghost~60008d26d73408ce4ff8d1b7

Can someone talk to me about OpenTelemetry as a sidecar, mixed with prometheus receivers (for scraping the container)? I'm losing labels I'm trying to set in the sidecar. My architecture is, a workload deployed in kubernetes with an otel sidecar, the workload either pushes via opencensus/otlp to the sidecar, or the prometheus receiver on the sidecar scrapes the workload directly. Labels are applied under receivers.prometheus.config.global.external_labels. This sidecar then exports via OTLP to a central collector on the cluster that is scraped by a master prometheus instance. I get/can see the metrics, but my external_labels aren't being set. I'm expecting this isn't a supported configuration and the global block in the receivers is just ignored?

I'm interested to see either a working example using prometheus in this way, or a better way in general.

Tigran Najaryan

@tigrannajaryan

@/all if you are interested in logs please check out the proposal to contribute Stanza to OpenTelemetry with the goal of using it as the log collection library for OpenTelemetry Collector: open-telemetry/community#605

Ghost

@ghost~60008d26d73408ce4ff8d1b7

To my above question, nevermind. I figured out my labeling issue; the labels have to be applied under the static_configs block, the global.external_labels isn't honored here.

Joshua MacDonald

@jmacd

The @opentelemetry Metrics Workshop happens tomorrow at 9:30 PST at https://zoom.us/j/8203130519

9:30 - 9:45am PST: Opening remarks, organization of the day, how we got here, and the many streams of work. We have an API, language SDKs, a Protocol, a Collector, Receivers, Exporters, Semantic Conventions, and a connection to Tracing [organizer: OTel Committee]

9:45-10:15am PST: Community building: @opentelemetry has first-class @OpenMetricsIO support, and @PrometheusIO users are first-class users. These projects are meant to get along, and committed to it. [organizer: Alolita Sharma]

10:15 - 11:00am PST: @opentelemetry Collector deployment models (e.g., Daemonset vs. Statefulset), Agents, Sidecars, and first-class support for Prometheus Remote-Write [organizer: Jaana Dogan]

11:30 - 12:15pm The Metrics API and Data Model, how we integrated the @opencensusio feature set (Tracing and Metrics, combined!) with the @OpenMetricsIO and StatsD data models. Presenting the six @opentelemetry metric instruments [organizer: Bogdan Drutu]

12:15 - 13:00pm Histograms! How we’ll support high-resolution and variable-boundary histograms and the connection to sampling metric events [organizers: Josh MacDonald, Michael Gerstenhaber]

13:30 - 15:00pm Questions and answers

4 replies

Devi

@asldevi

hello, Is there anyone working on adaptive sampling as described in this issue or differently? jaegertracing/jaeger#365

7 replies

Stephen Hong

@stephenhong

Hello team, according to Resource SDK, a resource can be associated with the TracerProvider when the TracerProvider is created or it can be associated with the MeterProvider when the MeterProvider is created. So does this mean that an application would need to use both providers in order for the traces and metrics to be linked to each other? I'm understanding this as, by using both providers, the traces and metrics from an application share the same resource attributes and therefore, are able to be linked to each other. Am I correct to assume this?

5 replies

taman9333

@taman9333

Hello everyone I had a problem when installing OTEL with istio
I had set b3-propagator as here https://www.npmjs.com/package/@opentelemetry/propagator-b3
and it works but I get a warning in all my spans " invalid parent span IDs "?

5 replies

Jonatan Ivanov

@jonatan-ivanov

Are there plans to enable GH Discussions on https://github.com/open-telemetry/opentelemetry-specification too as it was enabled for other repos?

Ulrich Romahn

@uromahn

Quick question: are people/companies are using OTEL in production or is most of the work just PoC and explorations?

5 replies

Trying to figure out if we should go with OTEL or better stick with OpenTracing for now.

Not sure if this is the right conversation for this question ...

Naga

@tannaga

Hello .. I am writing a custom processor in otel collector. Is there a way to access the content in the request headers? I was looking to extract user agent, x-forwarded-for, etc info from the header

Juraci Paixão Kröhling

@jpkrohling

@tannaga there isn't a way to do it reliably, as an aggregating processor might be running before your processor, like the batch processor

1 reply

in other cases, you should have access to the HTTP (or only gRPC, not sure now) metadata contained in the context

paullly

@paullly

Hello guys, we are planning to use opentelemetry in our Program. I was wondering if there is anything like a weaver in order to add traces for every method. Is the Otel tracing designed to be used like that, or would you recommend opening spans in every method manual? Thanks

1 reply

Juraci Paixão Kröhling

@jpkrohling

it's typically an anti-pattern to trace all method calls -- you probably should use an instrumentation library to get data about your boundaries only (like incoming and outgoing HTTP calls )

if you need more info, then you instrument the parts that you need to understand better

paullly

@paullly

Thank you very much for your answer! I will look into it!

arjantop-cai

@arjantop-cai

Hi, I have trouble setting up prometheus kubelet/node scraping with otel-collector. I have this config:

- job_name: k8s/kubelet
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

This is what I see in logs:

2021-01-21T21:43:21.433Z    warn    internal/metricsbuilder.go:104    Failed to scrape Prometheus endpoint    {"component_kind": "receiver", "component_type": "prometheus", "component_name": "prometheus", "scrape_timestamp": 1611265401788, "target_labels": "map[beta_kubernetes_io_arch:amd64 beta_kubernetes_io_instance_type:k3s beta_kubernetes_io_os:linux instance:k3d-dev-server-0 job:k8s/kubelet k3s_io_hostname:k3d-dev-server-0 k3s_io_internal_ip:172.20.0.3 kubernetes_io_arch:amd64 kubernetes_io_hostname:k3d-dev-server-0 kubernetes_io_os:linux node_kubernetes_io_instance_type:k3s node_role_kubernetes_io_master:true]"}

Bhavesh Mistry

@bmistry13

I have OTEL instrumented via Spring Sleuth and OpenTenemetry agent
but traceid for Spring Sleuth is different from OpenTenemetry agent
How can I make traceid to be same so they both report same traceid

arjantop-cai

@arjantop-cai

Hi, I have trouble setting up prometheus kubelet/node scraping with otel-collector. I have this config:

- job_name: k8s/kubelet
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

This is what I see in logs:

2021-01-21T21:43:21.433Z    warn    internal/metricsbuilder.go:104    Failed to scrape Prometheus endpoint    {"component_kind": "receiver", "component_type": "prometheus", "component_name": "prometheus", "scrape_timestamp": 1611265401788, "target_labels": "map[beta_kubernetes_io_arch:amd64 beta_kubernetes_io_instance_type:k3s beta_kubernetes_io_os:linux instance:k3d-dev-server-0 job:k8s/kubelet k3s_io_hostname:k3d-dev-server-0 k3s_io_internal_ip:172.20.0.3 kubernetes_io_arch:amd64 kubernetes_io_hostname:k3d-dev-server-0 kubernetes_io_os:linux node_kubernetes_io_instance_type:k3s node_role_kubernetes_io_master:true]"}

Had to enable debug logging to see that the error is actually forbidden, my role was missing the node endpoint

1 reply

I feel like this should be logged at error log level

Poonai

@poonai

Anyone used tempo for tracing?

1 reply

Arsene

@Tochemey

Hello folks. I hope we are all doing well. I would like to know whether there is a good tutorial with gRPC.

7 replies

Eric Mustin

@ericmustin

:wave: is the contrib flavor of the otel-collector packaged in the aws cloudwatch agent now for ec2?

re: https://aws.amazon.com/about-aws/whats-new/2021/01/amazon-cloudwatch-agent-now-supports-opentelemetry-apis-and-sdks/

Stephen Hong

@stephenhong

Hello team, I'm searching for any resources for creating a custom exporter for OpenTelemetry collector. I'm trying to send traces and metrics data to a specific destination and I need to create a custom exporter from scratch. This exporter also needs to be able to work with the Otel collector. Are there any good starting points, docs, tutorials, etc?

4 replies

Morgan McLean

@mtwo

FYI all, I've updated all Zoom meetings to require a password: 77777. This is now embedded into our calendar and meeting notes links, and is referenced on the community GitHub README.md.

We had to do this after one of the meetings got Zoom bombed! If you access the meetings via the calendar, you won't notice any changes.

also, we've enabled GitHub discussions on the community repo: https://github.com/open-telemetry/community/discussions

if you want invites to SIG meetings (with the updated Zoom links) in your calendar (rather than opening a separate one), I recommend joining https://groups.google.com/forum/#!forum/opentelemetry-calendar

Ted Young

@tedsuo

/@all :point_up:

Emmanuel Courreges

@ecourreges-orange

Hi all, has anyone performed performance comparisons for a simple java tomcat api that calls one rest backend?
I am interested at getting a rough percentage impact of tracing at let's say 1% tracing and 100% tracing with default attributes, with either spring-cloud-sleuth, opentelemetry-java, and opentelemetry-java-instrumentation, the latter being the preferred choice if it is equivalent to the former two.
This will allow me recommending instrumentation choices and sampling probability for production apps.
I know you might answer "It depends on the complexity of your code", that's why here I am talking of an almost "pass-through" code, in which instrumenting server and client spans will have the most performance impact, so it's a worst case performance impact. Thanks.

1 reply

For example I did a test with jaeger tracing simple requests on an Apache server C++ module Rest API which does one call to cassandra, and the impact of 8 spans per request with 100% tracing was a 50% rise in CPU usage, not for production, but 1% tracing was less than 3% CPU compared to no instrumentation, so that is acceptable for production use

Stavros Kontopoulos

@skonto

Hi all I am exporting some metrics at the collector container side (port 8889) using the prometheus exporter there. However, I dont see any labels coming from my resource set correctly in the controller at my app. I do see the resource labels printed locally in stdout when the metric points are received (debug mod). Any ideas? Is this supported or resource labels are removed?

Weyert de Boer

@weyert

This easycla stuff is not easy at all

jaguar75

@jaguar75_twitter

Hi, is there a way I can get context propagation without really emitting traces?

28 replies

Weyert de Boer

@weyert

Does anyone know how to opentelemetry logging should work? Doesn't mean I would need to attach events to the current span when my logger class is calling log('message')?

Where communities thrive

People

Repo info

Activity