Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Nuutti Kotivuori
    @nakedible-p
    Am I missing something, or is this sort of half-way baked still?
    janxyz
    @janxyz:matrix.org
    [m]
    You can use with_current_context() to simply attach the currently active context. https://docs.rs/opentelemetry/0.13.0/opentelemetry/trace/trait.FutureExt.html#method.with_current_context
    Nuutti Kotivuori
    @nakedible-p
    Should the example be updated to reflect that? https://github.com/open-telemetry/opentelemetry-rust/blob/main/examples/async/src/main.rs - I'm still not sure how that makes things easier - and it still doesn't seem to be handling setting the error response in to the span.
    (afk a bit)
    janxyz
    @janxyz:matrix.org
    [m]
    I understood from your previous post that you find it hard to extract the span context from a running process and forwarding it to an asynchronous function. Was I wrong with that assumption?
    Nuutti Kotivuori
    @nakedible-p
    Guess what I'm looking for is more what tracing provides in their instrument method - that a span is automatically entered on that async call, and exited when done, and status and exception in the span is set automatically based on the results of that function call. To easily track how long it takes for an async function to complete, and if the result was success or error.
    janxyz
    @janxyz:matrix.org
    [m]
    Something like with_span(span_name, fn) that takes a closure or something similar and traces it?
    Nuutti Kotivuori
    @nakedible-p
    I'd like to be able to write TcpStream::connect(proxy_addr).with_span("connect").await?; inside an async function - and the result would be that there would be a "connect" span which is entered when that call starts, and exited when that call finishes, and if the Result from that call is Err, then the Err is recorded in to the span - and if the Result is Ok, then 200 OK is recorded as the status.
    And the span would automatically have as parent the currently active span, as is customary
    janxyz
    @janxyz:matrix.org
    [m]
    Sounds nice! I don't think that's in the library yet though so you'd need to role your own 😬 but I might be mistaken
    Nuutti Kotivuori
    @nakedible-p
    That sounds like it should be quite a basic feature... all in all it would feel like opentelemetry-rust has a lot of implementation done on the internal pieces, but the actual consumer side is kinda clunky. For example, the fact that I usually need to create a tracer object, then start a span inside that tracer object, and then set that span active in the context, all on separate lines is awfully spammy. I'm willing to maybe spend 1 line in a function to ensure it is traced, not spend more lines on handling tracing than actual implementation.
    Anyway, will need to think on this and maybe write some helper of my own.
    Nuutti Kotivuori
    @nakedible-p
    Thank you for the help!
    janxyz
    @janxyz:matrix.org
    [m]

    That can all be written in a single line with

    global::tracer("my-component").start("span-name");

    And there are also other quality of life functions like in_span which executes a closure the same way you asked for it before:
    https://docs.rs/opentelemetry/0.13.0/opentelemetry/trace/trait.Tracer.html#method.in_span

        global::tracer("my-component").in_span("span-name", |_cx| {
            // anything happening in functions we call can still access the active span...
            my_other_function();
        })

    I don't think it is fair to the maintainers to complain about the maturity of the library in a 0.13.0 pre-release version. If you are lacking functions then I am sure they would appreciate a contribution or issue to see if others are also interested in such a function.

    Zhongyang Wu
    @TommyCpp
    Thanks for the feedback. Here are my two cents around the issue. So the reason why we have so many different "internal pieces" defined and exposed is that we have to comply with the opentelemetry specifications. I think many users are currently leverage tracing-opentelemetry to have a nicer customer interface. Feel free to open a feature request and we can have some discussion there.
    Nuutti Kotivuori
    @nakedible-p
    To be clear - I wasn't complaining as in the sense that "this should be better by now!" - I was complaining in the sense of trying to highlight what might be missing as a form of feedback. For me, trying to use the library has been really obtuse, as not a lot has been explained from the viewpoint of library user - even the documentation mostly explains the rules from the opentelemetry specifications rather than what the user needs. I will certainly try to make convenience wrappers myself, and see if I come up with something useful to contribute - or open a discussion in a ticket once there's something concrete to discuss.
    I've also used tracing-opentelemetry for a bit - and while that is much more geared towards user convenience, many parts of the whole tracing ecosystem rubbed me wrong, and I hate the additional complexity of chaining one framework to another.
    Nuutti Kotivuori
    @nakedible-p
    But, please don't take anything I say as entitlement on my part or discouragement to you - I'm just trying to accurately convey what I'm seeing as a user of the framework, rather than a builder.
    Zhongyang Wu
    @TommyCpp
    No worries :grimacing: I think it's nice to have some user feedback especially when we are talking about how to make this library easier to use.
    Nuutti Kotivuori
    @nakedible-p
    This is me not understanding opentelemetry, but what is the purpose of the tracer name? I mean, I can use global::tracer("foo") to create a named tracer by the name of "foo", but what I'm seeing is that it really sets "foo" as the name of instrumentatio library! That looks really odd to me - if that's the case, then it seems more that tracer objects should be created just once per application as a singleton, or something?
    New Tracer instances are always created through a TracerProvider (see API). The name and version arguments supplied to the TracerProvider must be used to create an InstrumentationLibrary instance which is stored on the created Tracer.
    And indeed, install_simple() on a pipeline does let tracer = provider.get_tracer("opentelemetry", Some(env!("CARGO_PKG_VERSION"))); and then returns that tracer, as if to signal that this is the tracer instance that should be used for all spans in the application/library
    But this is totally at odds with the usage in https://github.com/open-telemetry/opentelemetry-rust/blob/main/examples/async/src/main.rs for example
    I mean there there's a new tracer for every function
    async fn run(addr: &SocketAddr) -> io::Result<usize> {
        let tracer = global::tracer("runner");
        let span = tracer.start(format!("running: {}", addr));
        let cx = Context::current_with_span(span);
    
        let mut stream = connect(addr).with_context(cx.clone()).await?;
        write(&mut stream).with_context(cx).await
    }
    Nuutti Kotivuori
    @nakedible-p
    ...but since you can't define a "default tracer" (only default tracer provider) and you can't get a "current tracer" from context, the tracer instance would need to be either "global" or be passed on to each function separately. Passing it on to each function separately completely destroys the point of Context, so global would make sense - but it doesn't rub me right either
    I'd have to do something like:
    lazy_static! {
        static ref TRACER: GenericTracer = stdout::new_pipeline()
            .with_trace_config(
                sdktrace::config()
                    .with_sampler(sdktrace::Sampler::AlwaysOn)
                    .with_id_generator(sdktrace::XrayIdGenerator::default()),
            )
            .install_simple();
    }
    And even in that - it's fine for the application, but what should I do in a library?
    Nuutti Kotivuori
    @nakedible-p
    JS docs say something like this:
    const api = require("@opentelemetry/api");
    
    const tracer = api.trace.getTracer("my-library-name", "0.2.3");
    
    async function doSomething() {
      const span = tracer.startSpan("doSomething");
      try {
        const result = await doSomethingElse();
        span.end();
        return result;
      } catch (err) {
        span.setStatus({
          // use an appropriate status code here
          code: api.SpanStatusCode.ERROR,
          message: err.message,
        });
        span.end();
        return null;
      }
    }
    It still doesn't make sense to say that it is an instrumentation library - but maybe that's just a weirdness of opentelemetry that should be explained. I mean sure, it makes sense if all the spans are created by auto-instrumentation, but it's hella weird if writing spans manually.
    But on the rust side, that would mean I'd stick this everywhere:
    lazy_static! {
        static ref TRACER: GenericTracer = global::tracer(env!("CARGO_PKG_NAME"), Some(env!("CARGO_PKG_VERSION"));
    }
    Nuutti Kotivuori
    @nakedible-p
        /// Sets the status of the `Span`. If used, this will override the default `Span`
        /// status, which is `Unset`. `message` MUST be ignored when the status is `OK` or `Unset`
        fn set_status(&self, code: StatusCode, message: String) {
            self.with_data(|data| {
                if code == StatusCode::Error {
                    data.status_message = message;
                }
                data.status_code = code;
            });
        }
    Doesn't that mean that it's possible to get a non-empty status_message for status OK by first setting status error with message, and then setting status OK? I mean, that's what the spec says, but I think what they mean is that status_message should be cleared if status is OK or Unset... or who knows
    Nuutti Kotivuori
    @nakedible-p
    Man am I bad at Rust... but this sorta works how I think it should work, although the calling convention is not up to snuff:
    async fn fut_with_span<T, Fut>(fut: Fut, tracer_name: &'static str, span_name: &'static str) -> Result<T, Error>
    where
        Fut: Future<Output = Result<T, Error>>
    {
        let tracer = global::tracer(tracer_name);
        let span = tracer.start(span_name);
        let cx = Context::current_with_span(span);
        let spanref = cx.span();
        let ret = fut.with_context(cx.clone()).await;
        match ret {
            Ok(v) => {
                spanref.set_status(StatusCode::Ok, "".to_string());
                Ok(v)
            },
            Err(e) => {
                spanref.record_exception(&e);
                spanref.set_status(StatusCode::Error, e.to_string());
                Err(e)
            },
        }
    }
    Nuutti Kotivuori
    @nakedible-p
    Finally, with a lot of trial and error and help from Rust discord, I managed to transform the above to something I can actually call on a future. It requires the async-trait crate, though.
    #[async_trait]
    trait FutureSpan: Future {
        async fn with_span<T, E>(self, tracer_name: &'static str, span_name: &'static str) -> Result<T, E>
        where
            E: std::error::Error,
            Self: std::future::Future<Output = Result<T, E>> + Sized + Send,
        {
            let tracer = global::tracer(tracer_name);
            let span = tracer.start(span_name);
            let cx = Context::current_with_span(span);
            let spanref = cx.span();
            let ret = self.with_context(cx.clone()).await;
            match ret {
                Ok(v) => {
                    spanref.set_status(StatusCode::Ok, "".to_string());
                    Ok(v)
                },
                Err(e) => {
                    spanref.record_exception(&e);
                    spanref.set_status(StatusCode::Error, e.to_string());
                    Err(e)
                },
            }
        }
    }
    
    impl<T, E, F> FutureSpan for F
    where
      F: Future<Output = Result<T, E>>
    {}
    Example usage:
        let outbound_stream = TcpStream::connect(proxy_addr).with_span("process_connection", "connect").await?;
    Nuutti Kotivuori
    @nakedible-p
    Next up I'll probably need to make a macro or helper to do something like Context::current().span().set_attribute(Key::new("key2").string("hello")
    should be span_attr!("key2", "hello") or something :-)
    Nuutti Kotivuori
    @nakedible-p
    Does anyone have any feedback on the tracer name vs. instrumentation lib thing? It's one of the bigger things I don't really understand in this whole thing...
    9 replies
    Nuutti Kotivuori
    @nakedible-p
    Another question - how do I create a span that does not have the currently active span as parent? It seems if I do tracer.span_builder().start(), then as parent context is set to None, it will still default to currently active span - which is kinda weird, as tracer.start() will especially use the method that will set Some(Context::current) as the parent context - but if the value is left as None, then the trace implementation does the same as well.
    3 replies
    bbigras
    @bbigras:matrix.org
    [m]
    Does opentelemetry-prometheus export metrics for all the span I'm using for opentelemetry?
    bbigras
    @bbigras:matrix.org
    [m]
    I'm guessing it's not the case.
    Julian Tescher
    @jtescher
    no you have to use the otel metrics api for that
    Nuutti Kotivuori
    @nakedible-p

    In AWS XRay, spans are often exported twice. First soon after creation, with in_progress: true, and then a second time after the span is ended. It is mandated that a span can only be delivered either once, when it is ended, or twice, first with in_progress true and second with in_progress false.

    Is there any mechanism in OpenTelemetry to export in-progress spans? I know a SpanProcessor does get start and end events, but it's not an Exporter.

    1 reply
    Zhongyang Wu
    @TommyCpp
    You can call exporter from the span processor though. You will have to implement your own span processor to do that
    Nuutti Kotivuori
    @nakedible-p
    Right, kind of figured that out myself - Exporter doesn't really care if they are in progress or complete - but I need to write my own SpanProcessor that will also export spans on start. The only snag-ish in that is that the end method gets a SpanData which is taken in by the Exporters, but start only gets a Span and Context, so I need to compose SpanData myself if I want to use that.
    Noel Campbell
    @nlcamp

    What's the reason behind having the interval parameter in opentelemetry_otlp::new_metrics_pipeline() and not just relying on the specified or default period value to control how often metrics are pushed? https://docs.rs/opentelemetry-otlp/0.5.0/opentelemetry_otlp/fn.new_metrics_pipeline.html

    Is there a known use case where someone would want to specify an interval other than the one seen in all the metrics examples? :

    // Skip first immediate tick from tokio, not needed for async_std.
    fn delayed_interval(duration: Duration) -> impl Stream<Item = tokio::time::Instant> {
        opentelemetry::util::tokio_interval_stream(duration).skip(1)
    }
    1 reply
    Giselle Serate
    @giselleserate-okta
    Hi all, I've been trying to get udp messages sent to an agent (instead of a collector) from a jaeger pipeline. I believe that when the max packet size is decreased, too-big packets just aren't getting sent, since when I watch the packets in wireshark, fewer packets come through with a smaller max packet size (as opposed to more, if they were being split up or something). Is this a bug, or an intended behavior?
    2 replies
    Nuutti Kotivuori
    @nakedible-p

    This is more of a OpenTelemetry question than related to Rust - but looking at the opentelemetry protobuf definition, then Span "attributes" is a list of KeyValues. And KeyValues always have a string key, and AnyValue as value. And AnyValue has can contain arrays and key value lists itself, meaning that a KeyValue list can easily be mapped from an object with arbitrary nesting - it's like JSON, except that there's also "int64" and "bytes" datatypes.

    However, when looking at the specification, it says that Span attributes can contain primitive types and homogenous arrays of primitive types only - so no nesting allowed.

    So two questions:
    1) Why is the attributes limited to primitive values in the spec even though the protobuf allows for more?
    2) Is it customary to just use encoded JSON as the value to overcome this limitation in all the cases where nesting required?

    2 replies
    Josh Triplett
    @joshtriplett
    I'm currently using tracing in my server, with text output at the moment, and I'm looking to start using tracing-opentelemetry and a remote service (possibly Honeycomb) for observability. My main server can just use telemetry-otlp and its batch mechanism directly, but that main server spawns various short-lived servers (separate instances) that may only run for seconds, and those short-lived servers can't afford to make multi-millisecond API calls to a remote service. I think it would make sense to have my main server accept telemetry from the short-lived servers, and upload it on their behalf in batch. I'm trying to figure out what people normally use for that; what Rust crates do I want to look into for accepting telemetry data and passing it on via the same OLTP connection I'm already using?
    Josh Triplett
    @joshtriplett
    The short-lived servers can count on the main server being continuously available, and any work done on the short-lived servers to get tracing data to the long-running main server should be as lightweight as possible.
    I don't know if I should be looking for a Rust in-process OpenTelemetry Collector to run on the main server, or if there's some much more lightweight mechanism I can use to get trace information from the short-lived servers to the main server and then let the main server handle exporting it via OTLP.
    9 replies