Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community

    Hi, I have trouble setting up prometheus kubelet/node scraping with otel-collector. I have this config:

    - job_name: k8s/kubelet
        scheme: https
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          - role: node
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)

    This is what I see in logs:

    2021-01-21T21:43:21.433Z    warn    internal/metricsbuilder.go:104    Failed to scrape Prometheus endpoint    {"component_kind": "receiver", "component_type": "prometheus", "component_name": "prometheus", "scrape_timestamp": 1611265401788, "target_labels": "map[beta_kubernetes_io_arch:amd64 beta_kubernetes_io_instance_type:k3s beta_kubernetes_io_os:linux instance:k3d-dev-server-0 job:k8s/kubelet k3s_io_hostname:k3d-dev-server-0 k3s_io_internal_ip: kubernetes_io_arch:amd64 kubernetes_io_hostname:k3d-dev-server-0 kubernetes_io_os:linux node_kubernetes_io_instance_type:k3s node_role_kubernetes_io_master:true]"}

    Had to enable debug logging to see that the error is actually forbidden, my role was missing the node endpoint

    1 reply
    I feel like this should be logged at error log level
    Anyone used tempo for tracing?
    1 reply
    Hello folks. I hope we are all doing well. I would like to know whether there is a good tutorial with gRPC.
    7 replies
    Eric Mustin
    :wave: is the contrib flavor of the otel-collector packaged in the aws cloudwatch agent now for ec2?
    Stephen Hong
    Hello team, I'm searching for any resources for creating a custom exporter for OpenTelemetry collector. I'm trying to send traces and metrics data to a specific destination and I need to create a custom exporter from scratch. This exporter also needs to be able to work with the Otel collector. Are there any good starting points, docs, tutorials, etc?
    4 replies
    Morgan McLean

    FYI all, I've updated all Zoom meetings to require a password: 77777. This is now embedded into our calendar and meeting notes links, and is referenced on the community GitHub README.md.

    We had to do this after one of the meetings got Zoom bombed! If you access the meetings via the calendar, you won't notice any changes.

    also, we've enabled GitHub discussions on the community repo: https://github.com/open-telemetry/community/discussions
    if you want invites to SIG meetings (with the updated Zoom links) in your calendar (rather than opening a separate one), I recommend joining https://groups.google.com/forum/#!forum/opentelemetry-calendar
    Ted Young
    /@all :point_up:
    Emmanuel Courreges
    Hi all, has anyone performed performance comparisons for a simple java tomcat api that calls one rest backend?
    I am interested at getting a rough percentage impact of tracing at let's say 1% tracing and 100% tracing with default attributes, with either spring-cloud-sleuth, opentelemetry-java, and opentelemetry-java-instrumentation, the latter being the preferred choice if it is equivalent to the former two.
    This will allow me recommending instrumentation choices and sampling probability for production apps.
    I know you might answer "It depends on the complexity of your code", that's why here I am talking of an almost "pass-through" code, in which instrumenting server and client spans will have the most performance impact, so it's a worst case performance impact. Thanks.
    1 reply
    For example I did a test with jaeger tracing simple requests on an Apache server C++ module Rest API which does one call to cassandra, and the impact of 8 spans per request with 100% tracing was a 50% rise in CPU usage, not for production, but 1% tracing was less than 3% CPU compared to no instrumentation, so that is acceptable for production use
    Stavros Kontopoulos
    Hi all I am exporting some metrics at the collector container side (port 8889) using the prometheus exporter there. However, I dont see any labels coming from my resource set correctly in the controller at my app. I do see the resource labels printed locally in stdout when the metric points are received (debug mod). Any ideas? Is this supported or resource labels are removed?
    Weyert de Boer
    This easycla stuff is not easy at all
    Hi, is there a way I can get context propagation without really emitting traces?
    28 replies
    Weyert de Boer
    Does anyone know how to opentelemetry logging should work? Doesn't mean I would need to attach events to the current span when my logger class is calling log('message')?
    think of trying to write an instrumentation for winston. I think what I am describing is happening in the log4j but my Java ain't great
    Bhavesh Mistry
    HI All, I would like to set attributes for all spans to have global attributes span.setAttribute("ui_Session", "session_uuid"); ? How do I achieve this with OTEL ?
    Bhavesh Mistry
    hi All, @WithSpan annotation is only applies to METHOD can we change behavior to applicable to Class to ALL methods will get instrumented it is very annoying to add for each method
    1 reply
    Maarten De Wispelaere
    Hey - We're thinking about integrating either opentracing or open-telemetry into several of our Java projects (and some NodeJS stuff). The latter (Open-Telemetry) seems like the way to go, but I wonder if it's stable enough at this time to use in production projects?
    Juraci Paixão Kröhling
    the tracing part as a whole has been deemed stable some time ago, and the java sdk is a "tier 1" sdk. At this point, I'd say that it's quite safe to use it in terms of API stability
    John Watson
    Just FYI, we uncovered a couple small API tweaks we need to make in the Java APIs before 1.0. But, they are small tweaks, and we always call out any breaking changes in our release notes and CHANGELOG.md with every release (and, once we hit 1.0, breaking changes are forbidden).
    Weyert de Boer
    anyone know if you can join the zoom meetings via the browser?
    5 replies
    Bryce Buchanan
    Hey @SergeyKanzhelev & @nachoBonafonte, would you all be open to starting a Swift SIG? I'd like to open a forum to discuss development of that project.
    1 reply
    Maarten De Wispelaere
    @jpkrohling @jkwatson Thanks guys!
    Patrick White
    Hello, I am working on a migration from opentracing to opentelemetry, I'm curious if there might be any issues that occur if I'm using the same cassandra cluster backend for opentracing collector and opentelemetry collector?
    I noticed two fields BufferPeriod and BufferCount in the Prometheus Receiver's config, but they look undocumented and unused. I'm a relatively noob in the space of Prometheus. Obviously these flags are intended to control buffering. But is the purpose simply reliability, improved throughput or such things? Or does will it allow for some sort of statefullness (aggregation, explicitly reporting "missing" metrics, etc.)?
    Can we use the aws open telemetry collector on prem
    1 reply
    Bartłomiej Płotka
    :wave: Hello Otel Community! Do we have anywhere, some high-level Open Telemetry goals listed somewhere? The closest things to scope/goals I found is https://opentelemetry.io/docs/concepts/what-is-opentelemetry/ - is there anything more official/expanded? Thanks! (:
    1 reply
    Stephen Hong
    Hello team, I'm getting the below error when I run make in opentelemetry-collector-contrib. May I ask for someone's help?
    addlicense FAILED => add License errors:
    /bin/sh: addlicense command not found
    make: *** [checklicense] Error 1
    14 replies
    Rashmi Modhwadia
    Hello Team, Anyone from new-relic opentelemetry-collector-contrib space and can please help me? I am not sure if its an account error or not?
    I have a basic setup with logging exporter and new-relic exoporter. Logging exporter writes my metrics on console but NR exporter fails with
    otel          |         {"component_kind": "exporter", "component_type": "newrelic", "component_name": "newrelic"}
    otel          | 2021-01-22T01:37:01.839Z        error   newrelicexporter/newrelic.go:47 2021/01/22 01:37:01.839525 {"err":"unexpected post response code: 403: Forbidden"}
    4 replies
    Will java code instrumented with micrometer work with open telemetry collectors? That is will there be support planned for an open telemetry metrics meterregistry
    1 reply
    Joan Porta
    Hi guys! in k8s, I want to use Opentelemetry collector to gatter logs, In the cluster I have multiple app's. Is it posible to not need in each app POD a sidecar with opentelemetry agent, just only the daemonset? I dont want extra overhead having to put a sidecar to all app's POD's
    1 reply
    Weyert de Boer
    I would like to talk through an idea here if it makes within the opentelemetry philosophy
    Weyert de Boer
    I am using a solution to do product analytics (e.g. google analytics or segment) and I was wondering if it would make sense to get those into Grafana for querying? I would assume I could write pull exporter for which the opentelemetry collector could use
    Weyert de Boer
    Anyone aware of a blurb that I could copy and paste in my design record about what OpenTelemetry goal and offers is?
    @flands @gramidt - Is there any guideline on how can I migrate from using vendor auto instrumentation to opentelemetry phase wise. I can not change all the applications in one go. There will be some applications with vendor specific tracing. New applications will start using opentelemetry tracing. I am not sure how this will come together.
    4 replies
    Anmol Dubey
    i want to disable db queries from the otel auto instrumentation. How can i do that ? I am using java agent for auto instruimenetation '''
    1 reply
    Hi all, took a good look around the website for this, didn't find anything. Curious what timeline exists for the first release, eyeing tracing specifically here. I remember it was the second half of 2020, have not seen an update. I manage observability at a firm and am hoping to use tracing for our java and javascript apps as we go deeper into a hybrid cloud scenario.
    3 replies
    This message was deleted
    Weyert de Boer
    anyone from google trace here?
    i am wondering if you are planning to support the opentelemetry semantic conventation labels for http
    15 replies
    CleanShot 2021-02-10 at 19.55.49@2x.png
    Weyert de Boer
    the baggage support in the trace spec, could that be used to share details about the request? E.g. I have a user id that I want to pass between services. Could I use this for it?
    6 replies
    Bogdan Drutu
    Weyert de Boer
    is there are a metrics room?
    2 replies
    Robert Pająk
    hello there
    can someone help me with EasyCLA, please?
    I try to follow this process: https://docs.linuxfoundation.org/lfx/easycla/contributors/corporate-contributor#github
    I got the approval from my manager (I got the email) but I am still stuck at the same step in the easycla wizard saying that Unfortunately, you are not yet approved by the Corporate CLA of Splunk, Inc. on Cloud Native Computing Foundation (CNCF).
    any ideas what is wrong?
    here are my PRs https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/pulls/pellared for reference
    1 reply
    Weyert de Boer
    @pellared yeah, i think your manager has to approve twice
    1 reply