Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    Zaheer Shaik
    Hi. The documentation for signalfxexporter says that it can only receive metadata updates from k8s_cluster receiver. I don’t see the receiver support in aws-otel-collector but I see the support for signalfxexporter. Could you pls share an example of if this exporter works with otlp metrics receiver?
    Min Xia

    Hi. The documentation for signalfxexporter says that it can only receive metadata updates from k8s_cluster receiver. I don’t see the receiver support in aws-otel-collector but I see the support for signalfxexporter. Could you pls share an example of if this exporter works with otlp metrics receiver?

    Have you tried the documentations here? - https://aws-otel.github.io/docs/partners/splunk#signalfx-exporter

    Nicolai Wendt
    Hi. I have set up ADOT with Lambda but am struggling with extending this to AWS Batch (ECS). I'm thinking of two approaches, but not sure if any of them are viable (a third option if you include using the latest cloud watch agent which should contain adot). I'd rather not pursue the first option of configuring all our jobs as multi-node parallel jobs communicating with a collector. Rather I'll keep attempting to deploy it as a deamon with every cluster (as that at least is supported for x-ray on aws batch). What approach would you recommend?
    2 replies
    I am new to AWS Distro - trying to research on OpenTelmetry - we are using node exporter to expose EC2 system metrics to Prometheus to pull. If we need to use AWS Distro to expose system metrics in OpemTelemetry format to be pulled by Prometheus - what needs to be done? Can someone please provide some links or information, that would be great, thanks!
    6 replies

    Hi, I was working on a use case where we have 2 fargate services which communicate via SQS. first services calls an API and publishes to SQS and then second one subscribes to the queue. It is java spring boot. we use the following code to subscribe:

    @SqsListener(value = "some-queue", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
        public void receive(@Headers Map<String, String> header, @Payload String message) {
            System.out.println("Header = " + header.get("AWSTraceHeader") );
            log.info("Incoming message: " + message);

    . But the service map that is creeated is not showing them as single graph.I can see the AWSTraceHeader having the right traceId. Can you please guide me on what I'm missing?

    Nicolas Venegas

    Hi, I’m having trouble getting my AWS Lambda JavaScript function to export to a locally running collector: My OPENTELEMETRY_COLLECTOR_CONFIG_FILE has an entry exporters.otlphttp.endpoint whose value is https://nvenegas-otel-collector.<tunnel-provider-domain>.com and my service.pipelines includes otlphttp as one of the exporters. That tunnel points to a locally running otel-collector from this demo: https://github.com/open-telemetry/opentelemetry-collector/tree/main/examples/demo

    Locally, I can export into that otel-collector instance via the tunnel’s URL, but when I run my lambda function I don’t even see any network calls to the tunnel URL.

    In the function logs I see this related to the exporter:

    INFO builder/exporters_builder.go:274 Exporter was built. {"kind": "exporter", "exporter": "otlphttp”}
    INFO service/service.go:200 Starting exporters…
    INFO builder/exporters_builder.go:92 Exporter is starting... {"kind": "exporter", "name": "otlphttp”}
    INFO builder/exporters_builder.go:97 Exporter started. {"kind": "exporter", "name": "otlphttp"}

    Have I misconfigured things?

    3 replies
    hostmetricsreceiver is not part of the https://github.com/aws-observability/aws-otel-collector/tree/b9d1519491ed1ff2d5f0abc5e546baa4eac291e2#aws-otel-collector-built-in-components aws-otel-collector yet! is it going to be supported as part of the aws-otel-distro soon for host metrics? otherwise what is the alternative?
    Michael Hyatt
    Hi there, I have a question re ADOT for Lamda. In X-Ray, I am seeing the additional "Initialization", "Invocation" and "Overhead" spans that are not reported to the embedded OTel collector. How can I get these additional spans reported? Here is my OTel collector config:
        loglevel: debug
        endpoint: xyz.com:1234
          receivers: [otlp]
          exporters: [logging, otlp]
          receivers: [otlp]
          exporters: [logging, otlp]
    4 replies
    Nicolas Venegas
    Do I need to instantatiate a BasicTraceProvider() in my Lambda (JavaScript) function and register span processors, e.g., CollectorTraceExporter to get tracing to work?
    That’s the only way I’ve thus far been able to get the collector layer to export outside of the lambda (see my question, above, from Jun 11th)
    Michael Hyatt
    Hi there. I am using ADOT for Lambda and want to trace 2 lambda invocations through SQS. I am not doing any instrumentation within lambdas themselves relying only on what Python SDK can autodetect and it is working quite well since I am using the boto package. I can see the OTel spans happening, but they are not being reported as part of the same trace and I want to do a distributed tracing that involves 2 lamdas calling each other through the SQS queue. Can someone please point me to a recipe on how to propagate the trace context through SQS?
    3 replies
    Dane Wilson
    I've setup the sample app for the opentelemetry-lambda on Nodejs using the AWS OTEL Lambda layer, however, I'm unable to get the AWS Instrumentation to work correctly. Currently I'm receiving the expected traces in X-Ray, apart from the S3 service spans.
    As part of exploratory I even tried manually registering the OTEL aws-sdk instrumentation but ended up with the same result.
    Any help would be much appreciated, cheers!
    Dane Wilson
    Dane Wilson
    Deleting my stack, renaming it, and then recreating it solved my issue.
    Michael Hyatt
    Hey there. I am using ADOT for Lambda with node, python and Java lambdas. I can only see metrics being sent with Java runtimes. Are metrics supported for python and node? Also, is there a way to send the lambda insights metrics using OTel collector as well?
    Brooke Sargent
    Hi! I am trying to set up the OTel Collector on EKS to receive from X-Ray and send to Honeycomb. I have it configured to where it says it's ready to receive in the logs and I have traces in X-Ray to but I don't think the listener is actually picking anything up. Am I missing anything in my YAML configuration to get this working? https://gist.github.com/brookesargent/0558ab8329a5f1db7bab52fbeeade036
    1 reply
    Can anyone help me adding additional Conig for SQS in AwsInstrumentation
    Michael Hyatt
    I noticed that Java Lambda instructions can handle custom HTTP context propagation with AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-proxy-handler. Other Lambda runtimes (Python, nodejs, etc) don't support this option. Can someone clarify please if the same option AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-proxy-handler is available for runtimes other than Java?
    2 replies
    Christopher Kruse (wear a mask!) 😷

    Hello! I'm trying to get the ADOT NodeJS Lambda Layer to work, and I've got a request coming in from API Gateway. The collector shows on an event:

    2021/07/07 18:27:48 [collector] Received event: {
        "eventType": "INVOKE",
        "deadlineMs": 1625682488966,
        "requestId": "[uuid]",
        "invokedFunctionArn": "arn:aws:lambda:[rest of ARN]",
        "tracing": {
        "type": "X-Amzn-Trace-Id",
        "value": "Root=[redacted];Parent=[redacted];Sampled=1"

    Yet, when the Lambda function itself runs, event.headers shows a reduced version of the Trace ID header, without the parent or sampled keys:

          'X-Amzn-Trace-Id': 'Root=[redacted]',

    Since the Trace ID isn't complete, it's never added to the Context, and any custom spans create new traces in Xray. I'd prefer to have them attached to the execution as a whole. Any ideas?

    1 reply
    Michael Hyatt
    I noticed the new Lambda layer for JS has been released - 0.23 and wondering if there are working examples that use it. I tried to follow the instructions here https://aws-otel.github.io/docs/getting-started/lambda/lambda-js and couldn't get the same example I had working with 0.19 version to work. My example is using OTEL_PROPAGATORS="tracecontext" setting.
    1 reply
    Yi Cheng
    Hello, is there anyone trying sending events from opentelemetry-cpp/examples/otl/ to AWS distro with otlp protocol? I tried to run it, but somehow never succeed. I'm wondering whether there is any workaround to make it work.
    2 replies
    Marcin Sodkiewicz
    Hi guys, after updating otel agent to latest version 1.4.1 it seems that metrics are no longer working. I am using java with aws-otel and integration with NewRelic
    5 replies
    Vincent Nmeregini

    Hello everyone glad to be here, I'm quite new to aws opentelemetry, I have been blocked trying to add the ARN of the Lambda Layer to my Lambda function. Kindly note the lambda function is built with Docker and by default it doesn’t support adding layers.

    Can someone please help me out


    Hello, has anyone been able to get the AWS Distro for Lambda working with Lightstep?

    I've followed the config pattern here: https://lightstep.com/blog/instrument-your-lambda-functions-with-opentelemetry-preview/

    When I use a config with processors and batch like the example, it complains "Error: cannot load configuration: unknown processors type "batch" for batch"

    We are having an issue where the otel agent is seeing spans collected from otlp and logging them but nothing is getting sent to xray and there are no logs suggesting errors. Any ideas how to troubleshoot? I have tried zpages and its not showing any errors either. We are using the aws java auto instrumentation for java client and the aws distro for the otel agent. Metrics are successfully being sent to emf but traces are not being sent to xray.
    1 reply
    Has anyone come acrross this issue with writting to AMP
    2021-08-26T19:17:41.683Z    error    exporterhelper/queued_retry.go:288    Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "name": "awsprometheusremotewrite", "error": "Permanent error: [Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination]", "dropped_items": 66}
    Driss Amri
    Does anyone have a working example of a Java AWS Lambda with ADOT?
    Submitted bug ticket from my post above: open-telemetry/opentelemetry-collector-contrib#4909
    If anyone is a familiar with using prometheus writter please take a look.
    Tim Fendt
    Hello everyone. I am attempting to setup the lambda layer with nodejs exporting to newrelic. I am getting the following error though:
    Error: cannot load configuration: unknown exporters type "newrelic" for newrelic
    I followed the docs and the collector.yaml matches. Is newrelic no longer supported or am I doing something else wrong? I did notice in the logs this line: [collector] Launching OpenTelemetry Lambda extension, version: v0.1.0 so I am curious if maybe the layer is using the wrong version? Looks like the latest version is 0.11.0?
    13 replies
    Vincent Nmeregini
    Hello, I'm using AWS Distro for OpenTelemetry Lambda arn:aws:lambda:eu-central-1:901920570463:layer:aws-otel-nodejs-ver-0-23-0:1, I already set AWS_LAMBDA_EXEC_WRAPPER to /opt/otel-handler.
    it's not tracing my database call, is this the expected behaviour or there's an issue somewhere?

    Hello, i am trying to get traces from kubernetes running on ec2's (installed with kOps) to AWS Xray. I have opened the permissions mentioned in https://aws-otel.github.io/docs/setup/permissions
    for my ec2's and installed ADOT operator + ADOT collector in deployment mode. The collector is configured to have otlp as receiver and awsxray as exporter. This collector receives the otlp data
    from another (standard) opentelemetry-collector. This standard collector also exports the data to a in-cluster jaeger instance where the traces are visible.

    The problem is that there are no traces in visible in xray (or any of the other views). I have tried debugging this problem using zPages and from there I can see that the configured
    otlp receiver does receive traces and awsxray exporter forwards these onwards. How could i debug this further or is the problem inherently in this kOps+ec2 setup not being supported?

    3 replies
    Phenix Rizen
    Hello, I'm using arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-collector-ver-0-36-0:1 lambda layer and trying to use datadog exporter. I'm getting this error Error: cannot load configuration: unknown exporters type "datadog" for datadog Am I missing something?
    Saef Taher
    Hey I am using the aws-otel-collector on ECS Fargate and noticed that the otel sidecar is constantly consuming more and more memory
    Anyone got similar issues?
    I'm in the process of integrating the AWS OTEL collector into an ECS Fargate service and using the X-Ray exporter. I have it all working, but would like to specify some friendly attribute(s) to better filter/select traces in X-Ray. The default uses http.url which isn't vary obvious to casual observers. Ideally I'd like to use resource.service.name as an indexed_attribute, but it appears that only the attributes metadata is index-able? Am I missing something? Is there a better way to do this? I'm coming from DataDog and the service name was fairly heavily used, but that doesn't seem to be the case with X-Ray so perhaps I'm not thinking about this correctly?
    5 replies
    Hey all, i'm trying to send metrics to cloudwatch, yet can make it work. I am using nodejs and using all the latest version of packages and aws-otel-collector. But i'm getting the following error "CollectorExporterError unknown field "doubleGauge" in v1.Metric" when opentelemetry exporter is truing to export metrics to collector. Any idea what's going on?
    Is it expected that that OTEL instrumentation in go does not also publish metrics related to published traces? Previously in datadog when a trace was published it also automatically published metrics related to the trace. (E.i http.request.hits with a count of requests, as well as the endpoint and response code of that request). Is this something that has to be manually instrumented? This seems like an easy win and most people would want metrics generated from traces as well.
    Jhonatan Forero Rosas
    Hi, is it possible to send traces with otlphttp exporter in format JSON than Protobuf#
    I tried with headers application/JSON but the results are the same, binary body
    Does anyone have a solution to getting automatic metrics generated from traces? This is something that datadog seemed to do out of the box and was wondering if there are any processors/recievers that will assist in auto generating metrics based off recieved traces?
    Tom Donnly
    Does anyone know of a metrics generator I can use to the v1/metrics endpoint? I am using the grpc receiver for metrics and want to do some testing
    Hi folks. i'm new to opentelemetry and having quite a time getting something useful to work. is there a good example somewhere showing how to get manual java tracing working, as described here: https://aws-otel.github.io/docs/getting-started/java-sdk/trace-manual-instr? would be even better if it was combined with lambda. all i really want to see are the traceIds showing in application logs
    How can I send the otel collection agents logs to a file when running in a docker container? I looked into the file exporter but that is much to verbose for what im looking for. I am primarly interested in the otel collectors startup logs. Is the best/only way to capture the stdout logs to use a docker log driver?
    Hi there folks, does anybody have the information on what is the latest lambda layer for python packages?
    It seems that arn:aws:lambda:REGION:901920570463:layer:aws-otel-python38-amd64-ver-1-9-1:1 does not have the batch processor, I am concerned that I will get rate-limited by my provider.
    Hungai Amuhinda

    Hi there. I’ve written a server in Golang, instrumented with otelmux, deployed to EKS with the aws-otel-collector sidecar. I am not able to see any metrics and when checking the logs for the otel collector I see:

    Exporting failed. Try enabling retry_on_failure config option.    {"kind": "exporter", "name": "awsxray", "error": "Permanent error: AccessDeniedException: \n\tstatus code: 403, request id: 8bdc1118-8aa1-4284-9559-f50cf42e3fe1"}

    I am following this guide https://aws-otel.github.io/docs/setup/eks. Has anyone seen something like this before?


    Hello everyone! I'm new to ADOT and facing some mostly configuration problems. I wonder if it's obvious to folks here what I missed.

    I'm trying to run a few apps (mostly auto-instrument) including collector on my laptop with docker, but send X-Ray data to a test cloud environment. Everything works fine for this "local" setup. I see what I expect in traces from CloudWatch console.

    However, when the apps are deployed to the cloud, and API gateway + ELB are used, I'm getting problems. First, there was no trace at all, but by using X-Ray propagator, that was solved. So that was a configuration issue.

    Now, the next problem is that I don't see stack trace (for some tests that cause 4xx errors or 5xx faults) in CloudWatch console when things are all running in the cloud. Running and collecting on my laptop shows stack trace.

    Is there some obvious (but I missed) configuration setting for collecting stack trace for 4xx/5xx errors/faults?

    1 reply
    Bharat Gadde

    Hello eveyone. I am using aws-otel-collector for exporting metrics using aws emf exporter.
    When my application pushes any new (metric (counter) + dimensions) combination for the first time, aws emf exporter is not uploading this metric to aws.
    But if the same combination is pushed for the second time, then it is getting uploaded to aws.
    I have tried with Prometheus exporter, prometheus exporter works fine.
    Please help me in solving this.

    May be related to open-telemetry/opentelemetry-collector-contrib#3425.

    Ruben Costa
    Hi everyone, is there any information regarding pushing logs from EKS Fargate nodes to OTEL collector? Will this be possible in the future?