Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    Samiur A

    Hi! I have deployed otel-collector on our cluster based on https://github.com/signalfx/splunk-otel-collector/blob/main/examples/kubernetes-yaml/splunk-otel-collector-gateway.yaml

    I'm getting this "append failed" msg, bit confused why is it happening!!

    otel-collector-8669588d78-qsj4l otel-collector {"level":"info","ts":1614648489.4920561,"caller":"scrape/scrape.go:930","msg":"","level":"warn","scrape_pool":"otel-collector","target":"http://localhost:8888/metrics","msg":"append failed","err":"HTTP 401 \"Unauthorized\""}

    Hello guys! just a little question
    is there any newer/current performance test result published somewhere?
    https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/performance.md seems to be bit old and there is also no performance.md in collector-contrib as well.
    Thanks and have a nice day! :)
    1 reply
    Eric Sirianni
    Hello! Is there a recommended best practice for how/where to set request-specific tags that we may want to be applied commonly across logs, traces and (in some cases) metrics? For static tags, we can use Resource but it's not clear to me what to use for request-specific tags (e.g. something like enduser.id). Baggage seems like it might be appropriate, but how do I then automatically propagate the baggage tags to the active Span?
    Supreet Deshpande
    While working on observability for a fleet of IoT devices, I came across many shared paradigms between IoT observability and openTelemetry. With some minor changes, openTelemetry could fit in for logging, traces, and metrics and provide a consistent model for IoT diagnostics too. Has anyone already taken a stab at such an idea?
    I would be glad to discuss it. (Hope I'm posting on the right forum.)
    Katty Polyak
    Is there a way to use files on disk to store context and later extract it to start a new span under the same trace id in OpenTelemetry python? Basically we have a main thread that receives HTTP requests from another service and writes the requests to a disk based queue. These are later picked up by worker threads and I'd like to continue the trace with a new span at this point. However, starting a new span with this context object {'traceparent': '00-2ae64dcf2b8dd14217a42f0cc358d5ce-16a8fc1b04a79a81-01'} just results in a brand new trace/span: SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False)
    Hi, I have just started looking to use the open telemetry log collector agent which used to be stanza. I am looking at the following repo for the same. https://github.com/open-telemetry/opentelemetry-log-collection. Is there any documentation on how to build it locally and create a local docker image for the same.
    1 reply
    Ryan Bartsch
    Hi all,
    I'm not trying to solve a particular problem or anything - just doing some research/discovery into Open Telemetry. One of my questions is can I capture trace data using a bytecode/MSIL instrumentation approach (like how DynaTrace OneAgent works, but ideally using an open-source approach/agent) and send this information to an Open Telemetry Collector agent?
    I've asked the question in a lot more detail on Stack Overflow - see https://stackoverflow.com/questions/66486827/bytecode-instrumentation-with-open-telemetry-collector-agent. Apologies in advance if linking to SO is not following the correct protocol on this board - just casting a wider net to hopefully answer my question :)
    10 replies
    Ignacio Hagopian
    Question: If I use a Float64ValueRecorder, is there a way to also specify the bucket sizes? I'm using a Prometheus exporter.
    David Oceans
    the pods are CrashLoopBackOff
    2021-03-08T08:33:05.129Z        info    service/service.go:411  Starting OpenTelemetry Collector...     {"Version": "v0.20.0", "GitHash": "a10a1a7a", "NumCPU": 2}
    2021-03-08T08:33:05.136Z        info    service/service.go:592  Using memory ballast    {"MiBs": 204}
    2021-03-08T08:33:05.136Z        info    service/service.go:255  Setting up own telemetry...
    2021-03-08T08:33:05.219Z        info    service/telemetry.go:102        Serving Prometheus metrics      {"address": "", "level": 0, "service.instance.id": "d3dd5ed8-a0d0-44df-b17a-185a4001420f"}
    2021-03-08T08:33:05.220Z        info    service/service.go:292  Loading configuration...
    2021-03-08T08:33:05.222Z        info    service/service.go:303  Applying configuration...
    2021-03-08T08:33:05.222Z        info    service/service.go:324  Starting extensions...
    2021-03-08T08:33:05.222Z        info    builder/extensions_builder.go:53        Extension is starting...        {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check"}
    2021-03-08T08:33:05.222Z        info    healthcheckextension/healthcheckextension.go:40 Starting health_check extension {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check", "config": {"TypeVal":"health_check","NameVal":"health_check","Port":13133}}
    2021-03-08T08:33:05.222Z        info    builder/extensions_builder.go:59        Extension started.      {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check"}
    2021-03-08T08:33:05.223Z        info    builder/exporters_builder.go:306        Exporter is enabled.    {"component_kind": "exporter", "exporter": "logging"}
    2021-03-08T08:33:05.223Z        info    service/service.go:339  Starting exporters...
    2021-03-08T08:33:05.223Z        info    builder/exporters_builder.go:92 Exporter is starting... {"component_kind": "exporter", "component_type": "logging", "component_name": "logging"}
    2021-03-08T08:33:05.223Z        info    builder/exporters_builder.go:97 Exporter started.       {"component_kind": "exporter", "component_type": "logging", "component_name": "logging"}
    Error: cannot setup pipelines: cannot build pipelines: error creating processor "memory_limiter" in pipeline "metrics": checkInterval must be greater than zero
    2021/03/08 08:33:05 application run finished with error: cannot setup pipelines: cannot build pipelines: error creating processor "memory_limiter" in pipeline "metrics": checkInterval must be greater than zero
    David Oceans

    Can you help me understand this? If I install the OpenTelemetry Collector do I need to have a jaeger agent running or is it enough that from the application's instrumentation (I use jaeger libraries) it sends it to the opentelemetry collector?

    And another question, if my application only sends traces of jaeger I can deactivate the receivers of prometheus and zipkin and leave only jaeger, right?

    thank you very much
    Sagar Raj
    Can someone share their experience using Open-Telemetry with Project Reactor? What data does Otel provide? Can I this be a replacement for micrometer?
    1 reply
    Mateus Silva
    Can I use github.com/jaegertracing/jaeger-client-python to communicate to an opentelemetry agent using receivers: jaeger ?
    Juraci Paixão Kröhling
    yes, when using thrift_compact protocol
    Mateus Silva
    @jpkrohling thanks, I will search for more information, I tried that but I can get any span on jaeger web interface.
    problem in go 1.16..
    ➜  git:(master) ✗ env GOPROXY=direct go get go.opentelemetry.io/otel/sdk@v0.18.0
    go: downloading go.opentelemetry.io/otel/sdk v0.18.0
    go get: module go.opentelemetry.io/otel@v0.18.0 found, but does not contain package go.opentelemetry.io/otel/sdk
    1 reply

    Hi, I’m newbee in opentelemetry . could I have get some examples for k8s processer?
    i mapped

      - from: resource_attribute
        name: k8s.pod.ip

    with extracted istio attributes, but couldn’t see other mapped k8s attributes.

    How do we link manual instrumentation traces with automatic instrumentation traces . They are not coming under same hierarchy
    Simon Tharby

    Hi. I'm new to open-telemetry. Trying to use https://github.com/garthk/opentelemetry_honeycomb#opentelemetryhoneycomb with an elixir app on a kubernetes cluster. Using a config, in config.exs of

    config :opentelemetry,
      processors: [
        otel_batch_processor: %{
             write_key: System.get_env("HONEYCOMB_WRITEKEY"), dataset: "api-telemetry"}

    Should I also be setting an :api_endpoint option? (seen in: https://hexdocs.pm/opentelemetry_honeycomb/OpenTelemetry.Honeycomb.Config.html#t:config_opt/0). Thanks!

    Ben Landrum
    Which C++ SDK would you recommend for production?
    3 replies
    Ryan Bartsch
    I really like the OTEL collector approach. I think it's a great leap forward in terms of mitigating vendor lock, so thanks to all involved in this project :)
    One thing that I was thinking about is that the company I work for could end up with quite a few OTEL collector agents in our ecosystem - even if we deploy the agent as more of a gateway component... FWIW the company I work for has 3000+ servers/hosts spanning multiple data centers & cloud providers.
    Is there a project for remote config of the OTEL agents e.g. some kind of remote control plane...
    We use Terraform & Ansible, but it could be a bit tricky to manage this in a lot of places - would be nicer to be able to do this centrally from single repo... especially given that the config/policies might vary depending on the host e.g. some we might only want metrics, others we might want logs, metrics, APM...
    What strategies would be suggested. Note: if config management tool like Ansible is the suggested approach, that's fine... just doing some discovery...
    Robert Pająk
    can anyone send me an invite to the GH organization, please? :)
    GH issue: open-telemetry/community#678
    Hey geeks I would like to know what I am missing to get this:
    latency_bucket{operation="",service_name="",span_kind="SPAN_KIND_INTERNAL",status_code="STATUS_CODE_UNSET",le="2"} 199
    and this my collector config:
                - job_name: 'xxxxxxxx'
                  scrape_interval: 5s
                    - targets: ["localhost:9103"]
              metrics_exporter: prometheus
             url: ${"${JAEGER_ENDPOINT}"}
            endpoint: ""
              receivers: [otlp]
              processors: [spanmetrics, batch]
              exporters: [jaeger_thrift]
              receivers: [otlp]
              processors: [batch]
              exporters: [prometheus]
          extensions: [health_check]
    John Watson
    The UNSET status code means "not an error"
    @jkwatson but how do we get the actual status code? Maybe I am really noob here
    1 reply
    Lanh Nguyen
    Hi, I am trying to export logs in .Net to Otlp but I didnt see the Exporter for that, only see Console
    Pankaj Ch
    Hi, I am a Product Manager and got to know about the Open-telemetry project recently and would like to contribute to the project. Is there a place where i can see list of non-dev related things that i can help with?
    Hi all! For those of you that use Elasticsearch, we published a new OpenTelemetry Elasticsearch instrumentation for Node.js.
    We’d love to get your feedback https://github.com/aspecto-io/opentelemetry-ext-js/tree/master/packages/instrumentation-elasticsearch
    2 replies
    Pankaj Ch
    Hi, I am PM and love the open telemetry project , i am putting together a plan for my product to start supporting and contributing. Can someone share a similar experience/challanges they faced when try to convince the rest of the org ?
    Ben Ye
    Hello! One question about spanmetrics processor. Seems it doesn't support exporting the generated metrics to Prometheus directly because of the unsanitized labels, does anyone encounter the same issue?
    Screenshot from 2021-03-23 19-16-49.png
    Ben Ye
    Nvm, it was fixed in the latest main branch.
    vadlamudi tejaswini
    Question on OpenTelemetry integration with Jaeger as a trace collection system
    If we use OpenTelemetry Client Libraries, do we need to send traces to the Jaeger agent or send it directly to the Jaeger collector?
    @iNikem is anyone interested in this: https://github.com/opentracing-contrib/java-reactor/pull/12/files being adapted for open-telemetry?
    2 replies
    Jt Whissel
    I am trying to get my logs / stack traces from my web javascript app to opentelemetry. How does one send logs to opentelemetry, I cant seem to get it to do it. What I want is to send logs through opentelemetry collector to loki
    2 replies
    Hey everyone. I was unable to find any good tutorials for beginners on how to securely send metrics to a collector from a frontend. If anyone can help me in that direction, it would be great. I would be happy to contribute a step by step tutorial for people like on Medium/docs if there isn't one. Thanks.
    @mtwo Were you able to implement this on Android/iOS? I am still looking for how to configure clients without env variables and securely. Still haven't found it.
    1 reply
    Zaheer Shaik

    Hi. Could somebody advice on how to setup “trace ID aware load balancing”? I’m trying to setup OTel collector cluster which can sample successful traces, but not sample a trace with atleast one erroneous spaan.

    I see this note on tail sampling processor repo “Technically, trace ID aware load balancing could be used to support multiple collector instances, but this configuration has not been tested. “.

    supriya mahajan
    Hi, can someone help me understand if an open-telemetry collector be deployed just as a gateway service for multiple applications? Or is it always mandatory to run the collector as an agent on the host container?
    supriya mahajan
    I keep reading that it is recommended to run collector as an agent on the host container and additionally we can run collector as a gateway service as well. It tells me that running the collector as an agent is a must and running the collector as a gateway service is optional?

    I keep reading that it is recommended to run collector as an agent on the host container and additionally we can run collector as a gateway service as well. It tells me that running the collector as an agent is a must and running the collector as a gateway service is optional?

    Design doc. Imo, if your all ready have old collector agents in system, like jaeger agents, probably run collector as standalone service is better, you needn’t change every applications

    I keep reading that it is recommended to run collector as an agent on the host container and additionally we can run collector as a gateway service as well. It tells me that running the collector as an agent is a must and running the collector as a gateway service is optional?

    Design doc. Imo, if your all ready have old collector agents in system, like jaeger agents, probably run collector as standalone service is better, you needn’t change every applications

    If you just introduct opentelemetry first time run collector as an agent maybe better, application can report data to nearest collector.

    Brad Frost

    Hey all, looking for some advice on how to structure spans that we have implemented at the transport layer as opposed to application layer. We have TCP load balancer in our edge network that manages incoming connections/certs. We recently implemented 3 different spans: one for the connection establishment (a parent span), another for fetching the cert (child), and finally one for how long to proxy to the corresponding backend service (child).

    Given the connection establishment span is at the TCP layer, this makes it difficult to extract context from a client side span since we don't have access to HTTP headers (at least in that moment). Ideally, our client side spans would be the parent of the connection establishment span.

    I'm wondering if this is possible, or if our spans should be setup differently. There aren't too many examples online of running distributed traces below the application layer.

    2 replies
    Hello, opentelemetry has been introduced as observability framework for cloud native software. Can opentelemetry also be used for client desktop applications (considering casses like being offline, buffering etc.)?
    2 replies
    Daniel Gorlovetsky
    Hey I was wondering if there are any plans to fix protobuf exporter support in client-side based opentelemetry-js implementations
    4 replies
    Kavindu Ariyasinghe

    Hi all, I'm trying to dynamically turn on/off traces in opentelemetry agent level. I used Dotel.traces.sampler.

       java -javaagent:opentelemetry-javaagent-all.jar
     -Dotel.traces.exporter=otlp \
     -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
     -Dotel.otlp.span.timeout=4000 \
     -Dotel.resource.attributes=service.name=pet-clinic \
     -Dotel.traces.sampler="always_off" \
     -jar target/spring-petclinic-2.4.5.jar

    But the problem is when I need to turn on(Dotel.traces.sampler="always_on") traces I should stop the process and rerun the command by changing the environment variable. It means I need to restart my application too. Is there any alternative solution to turn on/off traces without restarting the application.

    2 replies