open-telemetry/opentelemetry-rust

Michael Duane Mooring

@mikeumus

:eyes:

I saw this one, so you have to manually send to the Prometheus server, it's not part of PrometheusExporter().init(); ?

Julian Tescher

@jtescher

this is using the pull configuration, so you have to have something periodically pull the exported metrics from that url

Michael Duane Mooring

@mikeumus

okay got it :thumbsup:

Julian Tescher

@jtescher

can also see an actix example here https://github.com/open-telemetry/opentelemetry-rust/blob/main/examples/actix-http-tracing/src/main.rs

Michael Duane Mooring

@mikeumus

:eyes:

Julian Tescher

@jtescher

same idea though

Michael Duane Mooring

@mikeumus

this one's just pulling too?

I saw this one before as well

Zhongyang Wu

@TommyCpp

Prometheu by default is running on a pull model where prometheus instance check the metrics endpoint in a certain interval to get the metrics. Prometheus also support remote write, which is a push model, but we don't have a prometheus remote write exporter yet

Michael Duane Mooring

@mikeumus

oh I see.

Julian Tescher

@jtescher

You may be able to enable it with the push flag documented in https://docs.rs/prometheus/

Michael Duane Mooring

@mikeumus

so I have Prometheus already running in another container, I can tell it to just pull form my jager/all-in-one? 🤞

Zhongyang Wu

@TommyCpp

Not exactly. You should ask it to pull metrics from your application's http endpoint for metrics

Michael Duane Mooring

@mikeumus

Julian Tescher

@jtescher

or if you have an existing prometheus registry set up from the above link you can specify it via https://docs.rs/opentelemetry-prometheus/0.5.0/opentelemetry_prometheus/struct.ExporterBuilder.html#method.with_registry

Michael Duane Mooring

@mikeumus

@TommyCpp is that different from the /metrics endpoint here?
https://www.jaegertracing.io/docs/1.21/cli/#jaeger-all-in-one

Zhongyang Wu

@TommyCpp

Yeah I believe that endpoint gives you metrics about Jaeger itself. Not your application

Michael Duane Mooring

@mikeumus

oh okay

my architeture is MacOS client app -> jaeger/all-in-one, so I don't know if I want to setup a web server on the client app for Prometheus to try to hit

Zhongyang Wu

@TommyCpp

To expose your metrics from you application. You should have a similar /metrics endpoint and ask your prometheus to scrape it. opentelemetry-prometheus can help you setup the handler of /metrics endpoint

Michael Duane Mooring

@mikeumus

Zhongyang Wu

@TommyCpp

Hmmm it does seem more like a push model here. One solution could be use opentelemetry-otlp and convert it into prometheus format in opentelemetry-collector

Or you can try prometheus pushgateway

Michael Duane Mooring

@mikeumus

yeah, I'm going to be pushing the data off the client app one way or another, can't have Prometheus trying to get metrics directly from the client app

okay let me look into these options, thank you @TommyCpp and @jtescher 🙇🏻‍♂️

Zhongyang Wu

@TommyCpp

Yeah np. I believe there are lots of discussion and solutions from prometheus community on how to handle your situation. Good luck

Michael Duane Mooring

@mikeumus

Found this rust-prometheus push exmaple here:
https://github.com/tikv/rust-prometheus/blob/master/examples/example_push.rs

Michael Duane Mooring

@mikeumus

@TommyCpp @jtescher the pushgateway worked! Thanks again guys :smiley:

Julian Tescher

@jtescher

We haven't really done a perf pass, but some nice ideas in https://github.com/rustcc/RustChinaConf2020/blob/master/rustchinaconf2020/RustChinaConf2020-22.钟镇炽-《高性能%20Rust%20tracing%20库设计》.pdf

Zhongyang Wu

@TommyCpp

Yeah I looked minitrace-rust performance testing before

Julian Tescher

@jtescher

could be fun to see where the costs are

I'd guess areas like context cloning, upgrading the tracer's provider reference a bunch of times, etc

Zhongyang Wu

@TommyCpp

But their test is pretty simple IIRC. Just recursively create 100 spans

Julian Tescher

@jtescher

yeah I think their use case is services that have one request root, and then create somewhere on the order of 100 children when servicing the request

their bench is synchronous as I recall, most of the heavy lifting in the minitrace impl or the rustracing impl is rayon's sync channel impl

always good to take those benchmarks with a grain of salt

but still could be interesting to see how much overhead there is in creating a span with a hundredish children

Zhongyang Wu

@TommyCpp

And they tested against tracing-opentelemetry. I modified the test to test against opentelemetry and was able to get some performance improvement

But I guess it's expected as it reduced a layer of abstraction

Julian Tescher

@jtescher

A quick check seems like perf with no exporters is ok

fn criterion_benchmark(c: &mut Criterion) {
    c.bench_function("many_children", |b| {
        let tracer = sdktrace::TracerProvider::default().get_tracer("always-sample", None);
        b.iter(|| {
            fn dummy(tracer: &sdktrace::Tracer, cx: &opentelemetry::Context) {
                for _ in 0..99 {
                    tracer.start_with_context("child", cx.clone());
                }
            }

            tracer.in_span("root", |root| dummy(&tracer, &root));
        });
    });
}

many_children time: [35.839 us 36.414 us 37.007 us]

not actually exporting anything with those, the batch processor's std::sync::channel send is probably another ~12us

Julian Tescher

@jtescher

actually a little trickier to measure exporters with this style

#[derive(Debug)]
pub struct NoopExporter;

#[async_trait::async_trait]
impl opentelemetry::sdk::export::trace::SpanExporter for NoopExporter {
    async fn export(
        &mut self,
        _batch: Vec<opentelemetry::sdk::export::trace::SpanData>,
    ) -> opentelemetry::sdk::export::trace::ExportResult {
        Ok(())
    }
}

fn criterion_benchmark(c: &mut Criterion) {
    c.bench_function("many_children", |b| {
        let provider = sdktrace::TracerProvider::builder()
            .with_simple_exporter(NoopExporter)
            .build();
        let tracer = provider.get_tracer("always-sample", None);
        b.iter(|| {
            fn dummy(tracer: &sdktrace::Tracer, cx: &opentelemetry::Context) {
                for _ in 0..99 {
                    tracer.start_with_context("child", cx.clone());
                }
            }

            tracer.in_span("root", |root| dummy(&tracer, &root));
        });
    });
}

time: [98.073 us 98.287 us 98.521 us]

with async exporter

#[derive(Debug)]
pub struct NoopExporter;

#[async_trait::async_trait]
impl opentelemetry::sdk::export::trace::SpanExporter for NoopExporter {
    async fn export(
        &mut self,
        _batch: Vec<opentelemetry::sdk::export::trace::SpanData>,
    ) -> opentelemetry::sdk::export::trace::ExportResult {
        Ok(())
    }
}

fn criterion_benchmark(c: &mut Criterion) {
    c.bench_function("many_children", |b| {
        let rt = tokio::runtime::Runtime::new().unwrap();
        let _g = rt.enter();
        let provider = sdktrace::TracerProvider::builder()
            .with_exporter(NoopExporter)
            .build();
        let tracer = provider.get_tracer("always-sample", None);
        b.to_async(&rt).iter(|| async {
            fn dummy(tracer: &sdktrace::Tracer, cx: &opentelemetry::Context) {
                for _ in 0..99 {
                    tracer.start_with_context("child", cx.clone());
                }
            }

            tracer.in_span("root", |root| dummy(&tracer, &root));
        });
    });
}

many_children           time:   [202.30 us 203.93 us 205.56 us]
                        change: [+102.42% +104.51% +106.59%] (p = 0.00 < 0.05)

but would have to look at the throughput vs the latency here, especially once the exporter isn't a no op

Where communities thrive

People

Repo info

Activity