Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Nuutti Kotivuori
    @nakedible-p
    JS docs say something like this:
    const api = require("@opentelemetry/api");
    
    const tracer = api.trace.getTracer("my-library-name", "0.2.3");
    
    async function doSomething() {
      const span = tracer.startSpan("doSomething");
      try {
        const result = await doSomethingElse();
        span.end();
        return result;
      } catch (err) {
        span.setStatus({
          // use an appropriate status code here
          code: api.SpanStatusCode.ERROR,
          message: err.message,
        });
        span.end();
        return null;
      }
    }
    It still doesn't make sense to say that it is an instrumentation library - but maybe that's just a weirdness of opentelemetry that should be explained. I mean sure, it makes sense if all the spans are created by auto-instrumentation, but it's hella weird if writing spans manually.
    But on the rust side, that would mean I'd stick this everywhere:
    lazy_static! {
        static ref TRACER: GenericTracer = global::tracer(env!("CARGO_PKG_NAME"), Some(env!("CARGO_PKG_VERSION"));
    }
    Nuutti Kotivuori
    @nakedible-p
        /// Sets the status of the `Span`. If used, this will override the default `Span`
        /// status, which is `Unset`. `message` MUST be ignored when the status is `OK` or `Unset`
        fn set_status(&self, code: StatusCode, message: String) {
            self.with_data(|data| {
                if code == StatusCode::Error {
                    data.status_message = message;
                }
                data.status_code = code;
            });
        }
    Doesn't that mean that it's possible to get a non-empty status_message for status OK by first setting status error with message, and then setting status OK? I mean, that's what the spec says, but I think what they mean is that status_message should be cleared if status is OK or Unset... or who knows
    Nuutti Kotivuori
    @nakedible-p
    Man am I bad at Rust... but this sorta works how I think it should work, although the calling convention is not up to snuff:
    async fn fut_with_span<T, Fut>(fut: Fut, tracer_name: &'static str, span_name: &'static str) -> Result<T, Error>
    where
        Fut: Future<Output = Result<T, Error>>
    {
        let tracer = global::tracer(tracer_name);
        let span = tracer.start(span_name);
        let cx = Context::current_with_span(span);
        let spanref = cx.span();
        let ret = fut.with_context(cx.clone()).await;
        match ret {
            Ok(v) => {
                spanref.set_status(StatusCode::Ok, "".to_string());
                Ok(v)
            },
            Err(e) => {
                spanref.record_exception(&e);
                spanref.set_status(StatusCode::Error, e.to_string());
                Err(e)
            },
        }
    }
    Nuutti Kotivuori
    @nakedible-p
    Finally, with a lot of trial and error and help from Rust discord, I managed to transform the above to something I can actually call on a future. It requires the async-trait crate, though.
    #[async_trait]
    trait FutureSpan: Future {
        async fn with_span<T, E>(self, tracer_name: &'static str, span_name: &'static str) -> Result<T, E>
        where
            E: std::error::Error,
            Self: std::future::Future<Output = Result<T, E>> + Sized + Send,
        {
            let tracer = global::tracer(tracer_name);
            let span = tracer.start(span_name);
            let cx = Context::current_with_span(span);
            let spanref = cx.span();
            let ret = self.with_context(cx.clone()).await;
            match ret {
                Ok(v) => {
                    spanref.set_status(StatusCode::Ok, "".to_string());
                    Ok(v)
                },
                Err(e) => {
                    spanref.record_exception(&e);
                    spanref.set_status(StatusCode::Error, e.to_string());
                    Err(e)
                },
            }
        }
    }
    
    impl<T, E, F> FutureSpan for F
    where
      F: Future<Output = Result<T, E>>
    {}
    Example usage:
        let outbound_stream = TcpStream::connect(proxy_addr).with_span("process_connection", "connect").await?;
    Nuutti Kotivuori
    @nakedible-p
    Next up I'll probably need to make a macro or helper to do something like Context::current().span().set_attribute(Key::new("key2").string("hello")
    should be span_attr!("key2", "hello") or something :-)
    Nuutti Kotivuori
    @nakedible-p
    Does anyone have any feedback on the tracer name vs. instrumentation lib thing? It's one of the bigger things I don't really understand in this whole thing...
    9 replies
    Nuutti Kotivuori
    @nakedible-p
    Another question - how do I create a span that does not have the currently active span as parent? It seems if I do tracer.span_builder().start(), then as parent context is set to None, it will still default to currently active span - which is kinda weird, as tracer.start() will especially use the method that will set Some(Context::current) as the parent context - but if the value is left as None, then the trace implementation does the same as well.
    3 replies
    bbigras
    @bbigras:matrix.org
    [m]
    Does opentelemetry-prometheus export metrics for all the span I'm using for opentelemetry?
    bbigras
    @bbigras:matrix.org
    [m]
    I'm guessing it's not the case.
    Julian Tescher
    @jtescher
    no you have to use the otel metrics api for that
    Nuutti Kotivuori
    @nakedible-p

    In AWS XRay, spans are often exported twice. First soon after creation, with in_progress: true, and then a second time after the span is ended. It is mandated that a span can only be delivered either once, when it is ended, or twice, first with in_progress true and second with in_progress false.

    Is there any mechanism in OpenTelemetry to export in-progress spans? I know a SpanProcessor does get start and end events, but it's not an Exporter.

    Zhongyang Wu
    @TommyCpp
    You can call exporter from the span processor though. You will have to implement your own span processor to do that
    Nuutti Kotivuori
    @nakedible-p
    Right, kind of figured that out myself - Exporter doesn't really care if they are in progress or complete - but I need to write my own SpanProcessor that will also export spans on start. The only snag-ish in that is that the end method gets a SpanData which is taken in by the Exporters, but start only gets a Span and Context, so I need to compose SpanData myself if I want to use that.
    Noel Campbell
    @nlcamp

    What's the reason behind having the interval parameter in opentelemetry_otlp::new_metrics_pipeline() and not just relying on the specified or default period value to control how often metrics are pushed? https://docs.rs/opentelemetry-otlp/0.5.0/opentelemetry_otlp/fn.new_metrics_pipeline.html

    Is there a known use case where someone would want to specify an interval other than the one seen in all the metrics examples? :

    // Skip first immediate tick from tokio, not needed for async_std.
    fn delayed_interval(duration: Duration) -> impl Stream<Item = tokio::time::Instant> {
        opentelemetry::util::tokio_interval_stream(duration).skip(1)
    }
    1 reply
    Giselle Serate
    @giselleserate-okta
    Hi all, I've been trying to get udp messages sent to an agent (instead of a collector) from a jaeger pipeline. I believe that when the max packet size is decreased, too-big packets just aren't getting sent, since when I watch the packets in wireshark, fewer packets come through with a smaller max packet size (as opposed to more, if they were being split up or something). Is this a bug, or an intended behavior?
    2 replies
    Nuutti Kotivuori
    @nakedible-p

    This is more of a OpenTelemetry question than related to Rust - but looking at the opentelemetry protobuf definition, then Span "attributes" is a list of KeyValues. And KeyValues always have a string key, and AnyValue as value. And AnyValue has can contain arrays and key value lists itself, meaning that a KeyValue list can easily be mapped from an object with arbitrary nesting - it's like JSON, except that there's also "int64" and "bytes" datatypes.

    However, when looking at the specification, it says that Span attributes can contain primitive types and homogenous arrays of primitive types only - so no nesting allowed.

    So two questions:
    1) Why is the attributes limited to primitive values in the spec even though the protobuf allows for more?
    2) Is it customary to just use encoded JSON as the value to overcome this limitation in all the cases where nesting required?

    2 replies
    Josh Triplett
    @joshtriplett
    I'm currently using tracing in my server, with text output at the moment, and I'm looking to start using tracing-opentelemetry and a remote service (possibly Honeycomb) for observability. My main server can just use telemetry-otlp and its batch mechanism directly, but that main server spawns various short-lived servers (separate instances) that may only run for seconds, and those short-lived servers can't afford to make multi-millisecond API calls to a remote service. I think it would make sense to have my main server accept telemetry from the short-lived servers, and upload it on their behalf in batch. I'm trying to figure out what people normally use for that; what Rust crates do I want to look into for accepting telemetry data and passing it on via the same OLTP connection I'm already using?
    Josh Triplett
    @joshtriplett
    The short-lived servers can count on the main server being continuously available, and any work done on the short-lived servers to get tracing data to the long-running main server should be as lightweight as possible.
    I don't know if I should be looking for a Rust in-process OpenTelemetry Collector to run on the main server, or if there's some much more lightweight mechanism I can use to get trace information from the short-lived servers to the main server and then let the main server handle exporting it via OTLP.
    9 replies
    Nuutti Kotivuori
    @nakedible-p

    This may be a bit controversial - and I don't know if the OpenTelemetry specification specifically mandates that default TraceId:s must be random 128-bits - but have you considered changing the default TraceId generator to use ULIDs instead of being completely random?

    https://github.com/ulid/spec - tl;dr on the spec: 48-bit milliseconds since epoch + 80 bit random

    And I don't mean adopting the string representation, and other stuff - just that instead of 128-bit random the beginning would be a timestamp by default. I think this will "magically" make a lot of stuff perform better.

    4 replies
    Nuutti Kotivuori
    @nakedible-p
    I suck at Rust - is there a more reasonable way to do the conversion of span Attributes into JSON than this?
                if sd.attributes.len() > 0 {
                    let mut map = Map::with_capacity(sd.attributes.len());
                    for (k, v) in &sd.attributes {
                        map.insert(k.to_string(), match v {
                            Value::Bool(x) => x.to_owned().into(),
                            Value::I64(x) => x.to_owned().into(),
                            Value::F64(x) => x.to_owned().into(),
                            Value::String(x) => x.to_owned().into(),
                            Value::Array(Array::Bool(x)) => x.to_owned().into(),
                            Value::Array(Array::I64(x)) => x.to_owned().into(),
                            Value::Array(Array::F64(x)) => x.to_owned().into(),
                            Value::Array(Array::String(x)) => x.to_owned().into(),
                        });
                    }
                    msg["attributes"] = map.into();
                }
    2 replies
    Alessandro Re
    @akiross

    Hello, I'm trying to get some tracing and metric data into otel collector (on the same host, default port). I'm using tracing for tracing data, and otel for metric data. I managed to get traces into the collector, but apparently I'm failing sending it the metrics.

    The code I'm using is similar to this: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=822f60525e95b24ec0c81a7f25dfa35f

    What am I doing wrong?

    4 replies
    Nisheeth Barthwal
    @nbaztec

    Could someone point me to a documentation on what are the use cases for different meters, as specified here:
    https://docs.rs/opentelemetry/0.14.0/opentelemetry/metrics/index.html

    From examples it seems as if both gauges and histograms can be implemented as ValueRecorders But then there are other things in there like Observers, BoundedCounter, etc. finding it a bit hard to figure out how to simply record a single value like bytes and incoming_requests and the use-cases for other meters.

    Thanks

    1 reply
    Glade Miller
    @glademiller
    I'm trying to use actix-web-opentelemetry and get the trace id and span id in my own middleware. For everything I have tried I always get a 0 trace id and span id is there a way to do this. Additionally I would love for my own middleware to be able to add a span that is in the context of the span from the actix-web-opentelemetry middleware and haven't had any luck making that work.
    Glade Miller
    @glademiller
    Perhaps the question should be phrased as how can I get the a trace id or span id that isn't 0. At what point are those assigned?
    The context for this is I need to return a header with those values in a certain format
    Glade Miller
    @glademiller
    I've narrowed down my problem and it is that the span created by actix-web-opentelemetry is not the current span in my own middleware. I'll be looking at actix's code a bit to see if there is a way to make sure the middleware layer correctly.
    Nate Mara
    @nate-onesignal
    Hello :wave: is it possible to directly set the sampling flag on an existing root-level opentelemetry span context? I'm using opentelemetry with tracing-opentelemetry and want to be able to set certain traces as non-sampled depending on some runtime data.
    9 replies
    Nisheeth Barthwal
    @nbaztec
    Hello, is it necessary to store the reference to ValueObserver created via meter.u64_value_observer.init() in static scope or can it be safely ignored within an underlying function?
    Derek Leverenz
    @derekleverenz
    Hi folks. I noticed a method i was using, https://docs.rs/opentelemetry/0.13.0/opentelemetry/trace/trait.TraceContextExt.html#tymethod.remote_span_context has been removed in 0.14. Is there another way I should be retrieving the remote span context now?
    2 replies
    Alessandro Re
    @akiross

    Is it possible to build a custom aggregator selector using opentelemetry_otlp 0.6.0? new_metrics_pipeline returns a OtlpMetricPipelineBuilder<Selector, ...> but the with_aggregator_selector is bound to Selector and not to a different type, as it happens in version 0.7.0.

    I wanted to use this to get some histogram metrics, but I cannot find a workaround. Can I get histograms in version 0.6?

    2 replies
    Alessandro Re
    @akiross

    for some reason, I cannot get counter instruments to work: I've set up the otel collector to use stdout exporter, and when counter data is exported, I get this:

    2021-06-01T10:15:33.577Z        INFO    loggingexporter/logging_exporter.go:57  MetricsExporter {"#metrics": 1}
    2021-06-01T10:15:33.577Z        DEBUG   loggingexporter/logging_exporter.go:63  ResourceMetrics #0
    InstrumentationLibraryMetrics #0
    InstrumentationLibrary example
    Metric #0
    Descriptor:
         -> Name: example.even
         -> Description:
         -> Unit:
         -> DataType: DoubleSum
         -> IsMonotonic: true
         -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
    DoubleDataPoints #0
    StartTimestamp: 2021-06-01 10:15:13.557887018 +0000 UTC
    Timestamp: 2021-06-01 10:15:33.559754668 +0000 UTC
    Value: 0.000000

    that is, Value is always zero. This is how I'm using it: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=c81f22d4ae8c428688dddedde55315d9

    21 replies
    Jeroen Vankelecom
    @jeroen-vankelecom
    Hi, I hope my question is at the right place here... We would like to annotate our metrics with traceIDs (exemplars)... looking at the opentelemetry-otlp code, it seems already some structure is in place to store Exemplars, but I'm missing the bigger picture. Any idea if and when exemplars will be supported in OpenTelemetry? Any good reference to how accomplish this would be highly appreciated! Thx
    1 reply
    wise0wl
    @wise0wl:jobmachine.org
    [m]
    Hello! I have an actix http web service that I have instrumented with opentelemetry using actix-web-opentelemetry. Currently it produces prometheus metrics and exports traces with the jaeger exporter. This works, however to do proper correlation I need to have the trace ID in my logs as well. Do any of you have suggestions for instrumenting logging with span and trace id's included?
    2 replies
    justin0mcateer
    @justin0mcateer
    I was wondering if there are plans to add support for logging in the opentelemetry-rust implementation
    wise0wl
    @wise0wl:jobmachine.org
    [m]
    I've seen talk of plans, but it is as yet unimplemented
    Zhongyang Wu
    @TommyCpp
    Yes, we do have plans to add log support. Currently, our main focus is still on tracing size(and metrics once it's stabilized)
    wise0wl
    @wise0wl:jobmachine.org
    [m]
    That did expose things, thank you. The traces and spans id's are all blank---for whatever reason. I'm trying a few things.
    Adrien Jeser
    @ajeser_gitlab

    Hello world !

    I am trying to send traces to Jeager in a Wasm application.

    My code:

    use opentelemetry::global;
    use opentelemetry::sdk::trace::{self, Sampler};
    use opentelemetry::trace::Tracer;
    
    fn main() {
        console_error_panic_hook::set_once();
        global::set_text_map_propagator(opentelemetry_jaeger::Propagator::new());
        let tracer = opentelemetry_jaeger::new_pipeline()
            .with_collector_endpoint("http://localhost:6831")
            .with_service_name("my_app")
            .with_trace_config(trace::config().with_sampler(Sampler::AlwaysOn))
            .install_simple()
            .unwrap();
    
        tracer.in_span("doing_work", |cx| for i in 1..100 {});
    
        global::shutdown_tracer_provider(); // export remaining spans
    }

    The feature "wasm_collector_client" is active on opentelemetry-jaeger.

    No traces are transmitted on the network. Does anyone have an idea?

    1 reply
    Zhongyang Wu
    @TommyCpp
    Hmm that should work. Will take a look today
    Kirill Mironov
    @Drevoed

    Hello everyone! For some reason I get message "unknown service opentelemetry.proto.collector.trace.v1.TraceService" when trying to send data to opentelemetry-collector.

    let tracer = opentelemetry_otlp::new_pipeline()
            .with_endpoint("http://localhost:55680")
            .with_tonic()
            .install_batch(opentelemetry::runtime::Tokio)?;
    
        let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);

    Cargo.toml:

    opentelemetry-otlp = "0.8.0"
    opentelemetry = { version = "0.15.0", features = ["rt-tokio"] }
    tracing-opentelemetry = "0.14.0"

    otel-collector-config.yaml:

    receivers:
      otlp:
        protocols:
          grpc:
    
    exporters:
      loki:
        endpoint: "http://loki:3100/loki/api/v1/push"
        tenant_id: "example"
        labels:
          attributes:
            severity: "severity"
    service:
      pipelines:
        logs:
          receivers: [ otlp ]
          exporters: [ loki ]

    docker-compose otel service:

      otel-collector:
        image: otel/opentelemetry-collector-contrib-dev
        command: [ "--config=/etc/otel-collector-config.yaml" ]
        volumes:
          - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
        ports:
        - "4317:4317"
        - "55680:55680"
        networks:
          - loki
    14 replies
    Otel-collector is up and running that's for sure:
    otel-collector_1  | 2021-07-11T18:33:45.879Z    info    builder/pipelines_builder.go:51 Pipeline is starting... {"pipeline_name": "logs", "pipeline_datatype": "logs"}
    otel-collector_1  | 2021-07-11T18:33:45.879Z    info    builder/pipelines_builder.go:62 Pipeline is started.    {"pipeline_name": "logs", "pipeline_datatype": "logs"}
    otel-collector_1  | 2021-07-11T18:33:45.879Z    info    service/service.go:192  Starting receivers...
    otel-collector_1  | 2021-07-11T18:33:45.879Z    info    builder/receivers_builder.go:70 Receiver is starting... {"kind": "receiver", "name": "otlp"}
    otel-collector_1  | 2021-07-11T18:33:45.879Z    info    otlpreceiver/otlp.go:75 Starting GRPC server on endpoint 0.0.0.0:4317   {"kind": "receiver", "name": "otlp"}
    bbigras
    @bbigras:matrix.org
    [m]
    In jaeger my spans are not linked (like with their parents). I'm using async code with the serenity discord lib which uses tokio and I use async-std in my code. Any ideas why the spans are not linked?
    Zhongyang Wu
    @TommyCpp
    Is the spans in the same host as the parent span or the parent span is in other hosts(like a remote span)?
    bbigras
    @bbigras:matrix.org
    [m]
    same host in the same program. I wonder if serenity does something special to run my async code. Or could a crate like tracing interfere with that?
    Zhongyang Wu
    @TommyCpp
    We do have a good integration with tracing so that should be OK. If it’s the same host the we can rule out propagator problems. Need to look at how serenity to have a better understanding