Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Kraig Amador
    @bigkraig
    well, the headers are there so if i am using a standard otel tracer, i should be able to get the trace id and parent id. the downstream requests wouldnt have the right parent spans (the synthetic span), but having the data in a trace is better than flying blind
    its absolutely a hack, but there isn’t a lot of options since x-ray is sampling. maybe we will get some good news before EOY on that one.. :)
    Cemalettin Koc
    @cemo
    I am trying to understand what aws-otel-distro is including apart from aws-otel-collector. Can someone clarify me please? Is there any other related tools included in aws-otel-distro?
    1 reply
    Michael Wilde
    @michaelwilde
    Hi. I noticed the distro has the xray receiver in it. Is the purpose here to flow all data thru the Otel Collector, and/or to make it possible to send telemetry to other destinations. the Xray-Segment-Format is fairly specific (and doesn't have a "duration" field). Is it possible to "xray data in, and non-xray tracing tool out (like Honeycomb, Jaeger, etc)
    1 reply
    Michael Hausenblas
    @mhausenblas
    I’m walking through https://aws-otel.github.io/docs/setup/eks and two notes:
    a. The example configuration template link (https://github.com/aws-observability/aws-otel-collector/blob/main/examples/eks/eks-sidecar.yaml) is broken, seems has been removed from the examples directory
    b. Given that the image is now available via https://gallery.ecr.aws/aws-observability/aws-otel-collector we don’t need to instruct folks to build the image themselves anymore or am I missing something?
    1 reply
    Michael Hausenblas
    @mhausenblas
    Just to confirm, seems to work fine (and maybe we want to add the EKS example back?):
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: adot-trace
      namespace: adot
    spec:
      selector:
        matchLabels:
          app: sample
      replicas: 1
      template:
        metadata:
          labels:
            app: sample
        spec:
          containers:
            - name: trace-emitter
              image: public.ecr.aws/g9c4k4i4/trace-emitter:1
              env:
              - name: OTEL_OTLP_ENDPOINT
                value: "localhost:55680"
              - name: OTEL_RESOURCE_ATTRIBUTES
                value: "service.namespace=AWSObservability,service.name=ADOTEmitService"
              - name: S3_REGION
                value: "eu-west-1"
              imagePullPolicy: Always
            - name: adot-collector
              image: public.ecr.aws/aws-observability/aws-otel-collector:latest
              env:
                - name: AWS_REGION
                  value: "eu-west-1"
    Mircea Cadariu
    @mcadariu_twitter
    Hi! We have two identical servers (using configuration-as-code), running Spring boot/JPA apps for which we want to use the distro and collector to send traces to AWS Xray. The curious situation I find myself in is that for one server, the entire setup works perfectly, but, the second one for which I can't yet identify any configuration drift, there is some glitch when it comes to extracting the DB interaction (e.g. it doesn't pick up the actual DB hostname but says it's "localhost" which it is clearly not, and also looking in Xray UI, I can indeed see "Repository" subsegments correctly however under them I'd expect to see child ones which that capture the actual SQL generated by Hibernate. I do see this info from the traces extracted from the other server. I can't yet see anything noteworthy in the collector/agent logs. Any pointers to further troubleshooting that I can do on the agent/collector side is appreciated, as mentioned, I can't yet spot a difference in the actual setup between these servers.
    Jamal Fanaian
    @jamal
    Hi, I am trying to try out the aws-otel-collector, migrating from our existing config. We are using the newrelic exporter, which from the documentation should be supported in the aws distro but when I load my config I am getting the following error:
    Error: cannot load configuration: unknown exporters type "newrelic" for newrelic
    4 replies
    Maor Goldberg
    @maorgoldberg
    Hey I have transitioned to the aws otel collector recently and was wondering if it is possible to tag traces with the relevant k8s resources like pods
    3 replies
    Cemalettin Koc
    @cemo
    Hi, I am trying to correlate AWS ALB Requests Logs with our apps tracing data. When will AWS start to expose OTEL compatible headers? 2. Can we get AWS managed services related spans as well at near future?
    Christopher Wong
    @christopher-wong

    Hi there. I’ve written two HTTP servers in Golang, instrumented with otelmux, deployed to EKS with the aws-otel-collector sidecar. In X-Ray, I see the traces coming in and the services within the service map, but the graph doesn’t look as expected. I have a service-a which calls an identical service-b, however I see a lone service-b bubble and a bunch of extra http.send/connect/headers noise.

    When service-a gets an inbound request, I then call service-b in the http handler to make an outbound request. That outbound request wasn’t getting linked to the inbound request from to service-a unless I manually call

    req, _ := http.NewRequest(http.MethodGet, "http://service-b.aws-otel-eks.svc.cluster.local/work", nil)
    ctx, req := otelhttptrace.W3C(r.Context(), req)
    otelhttptrace.Inject(ctx, req)

    although I was under the assumption that this should happen automatically as long as I had the correct propogators set, passing r.Context() directly without the need for Inject().

    My server config looks like this https://gist.github.com/christopher-wong/b1b3f149a4ec45196f7b85cbd760b52f#file-main-go

    Screen Shot 2021-01-06 at 11.37.00 AM.png
    2 replies
    Jean Jordaan
    @jean
    hi all -- how can we enable collectors from contrib (such as the elasticexporter)?
    Daniel Gomez Blanco
    @danielgblanco
    hi there, similar question here, is possible to configure the tracer to use other context propagators? (e.g. OtTracerPropagator )
    6 replies
    Mateusz "mat" Rumian
    @mat-rumian
    Hello everyone,
    I'm not sure but I think I found a bug/issue in the https://github.com/aws-observability/aws-otel-collector/blob/main/deployment-template/ec2/aws-otel-ec2-deployment-cfn.yaml . I was not able to use us-west-2 region so I had to select different one. I move to eu-central-1 (Frankfurt). In this case I had to use different AMI image (ami-03c3a7e4263fd998c). Becuase of that it wasn't possible to run AWS OTEL Collector. There was an error with directory or file does not exist. It came out that https://github.com/aws-observability/aws-otel-collector/blob/main/deployment-template/ec2/aws-otel-ec2-deployment-cfn.yaml#L130 is not present in the image. I had to add cd /home/ec2-user && wget https://aws-otel-collector.s3.amazonaws.com/amazon_linux/amd64/latest/aws-otel-collector.rpm to run it properly. Is it known issue? Or is it related to specific AMI image?
    1 reply
    Also could you please take a look into aws-otel/aws-otel.github.io#65 :)
    1 reply
    YossiCohn
    @yossicohn
    Hi,
    I get a regression with the Python SDK, it seems that I cannot get anything sent to AWS X-Ray, but I do see the metrics arrive at CloudWatch.
    I did see that the aws-otel-collector has not changed but the OpenTelemtry had bumped to 0.17b0 over Jan 21, 2021
    Does anyone get any regression?
    3 replies
    Zaheer Shaik
    @zaheershaik
    Hi - is there an planned release date for Node.JS lambda layer with OTEL support?
    2 replies
    Jean Jordaan
    @jean
    Hi, how would auth work? I saw this authenticationprocessor being referenced, but it isn't in master, and it looks like they're using oidc instead. Ah, I see that's explained in this blog post. Are there other approaches better suited to AWS?
    2 replies
    Robert Mills
    @rgmills
    Hey folks, I'm working on integrating the .NET otel sdk into our app and noticing interaction issues with that and AWS .NET SDK. Would this be a good place to discuss any issues or is there a better place?
    3 replies
    Jean Jordaan
    @jean
    Hi all .. does anyone know if there are OTel receivers for the Datadog browser-sdk?
    4 replies
    Bob
    @bviolier
    Hey all. I have been trying to get aws-otel to work with Fargate + ECS . So far most of the things are working. But I am running into two issues.
    1. it seems that the exporter awsemf/application as shown in the container-insight is not working properly. My aws-otel container logs show these errors, basically saying that I can't use these lognames, but also it couldn't find the "vars" to use.
    2. I've got the awsemf/performance to work by following this config, and the per-container metrics they do show up in the metrics explorer. And I can see task performance in the Container Insights console, but the "Container performance" stays empty. Should I maybe change some naming to make this work?
    3 replies
    Jean Jordaan
    @jean
    How would one go about exporting a subset of events to S3 ?
    Jean Jordaan
    @jean
    Has anyone created any additional authenticators? I'd like to use JWT, but without an OIDC server.
    Jean Jordaan
    @jean

    How would one go about exporting a subset of events to S3 ?

    It looks like Loki might work ...

    seetharamanis
    @seetharamanis
    Hi all, trying to install otel collector in local windows machine from here. https://aws-otel-collector.s3.amazonaws.com/windows/amd64/latest/aws-otel-collector.msi. Followed through the steps, but the service wouldn't start. Start-Service : Failed to start service 'AWS OTel Collector (AWSOTelCollector)'.
    At C:\Program Files\Amazon\AWSOTelCollector\aws-otel-collector-ctl.ps1:96 char:12
    2 replies
    Marcelo Olivas
    @mfolivas
    I am using the serverless framework along with Node. I would love to start using the hotel API. I tried to run the examples, but they did not work. Can someone point me where I can get examples on how to get started using telemetries on my lambda functions, standard logs? The only thing I noticed was some examples using python: https://github.com/aws-observability/aws-otel-lambda. I tried using the aws-otel-js but got this error: aws-observability/aws-otel-js#48
    niekname
    @niekname
    hey all, I am using the java auto instrumentation on my java / spring boot app which is running on ECS (fargate). I use the aws sidecar container v.0.7.0 and aws java agent 0.15.0. No custom configuration. In general it is working fine. But I have a problem with my service map. Next to the normal client > ECS:App > other dependencies, it also shows a lot of clutter. Mostly, client > UserRepository.findAll and a lot of other Spring repository classes. Do I need to change something in my configuration?
    Zaheer Shaik
    @zaheershaik
    Hi. is there a opentelemetry-aws-sdk for 1.x versions for aws-sdk Java?
    20 replies
    cathal
    @dannyboy84owl_twitter
    Hello everybody,
    Is it possible to export trace data to more than one aws xray account using the same collector? As far as I can tell, only one credentials file can be used which decids which account to push to.
    7 replies
    Zaheer Shaik
    @zaheershaik
    Hi. Is it required to sleep for a while before terminating the javascript process to ensure the trace records are flushed by the collector to the backend? I see this in the otel-js examples, but haven’t found similar references in aws-otel-js examples: https://github.com/open-telemetry/opentelemetry-js/blob/8fd139157d958ef9efbbddb3190cd0dd39f7be3e/examples/http/client.js#L29
    10 replies
    Lei Wang
    @wangzlei
    can you try spanProcessor.forceFulush() instead of waiting 5 sec?
    Marcelo Olivas
    @mfolivas

    I was able to configure the following counter:

    const { MeterProvider } = require('@opentelemetry/metrics')
    
    const { ConsoleLogger, LogLevel } = require('@opentelemetry/core')
    const { CollectorMetricExporter } = require('@opentelemetry/exporter-collector-grpc')
    
    
    const metricExporter = new CollectorMetricExporter({
        serviceName: 'aws-otel-js-sample',
        // logger: new ConsoleLogger(LogLevel.DEBUG),
        url: (process.env.OTEL_EXPORTER_OTLP_ENDPOINT) ? process.env.OTEL_EXPORTER_OTLP_ENDPOINT : "localhost:55680"
      })
    
    const meter = new MeterProvider({
        exporter: metricExporter,
        interval: 1000,
    }).getMeter('aws-otel-js')
    
    
    const requestCount = meter.createCounter('payload', {
        description: 'Metric for counting payload size'
    })
    
    const boundInstruments = new Map();
    
    module.exports.countAllRequest = () => {
        return (req, res, next) => {
            if (!boundInstruments.has(req.path)) {
                const labels = { stage: process.req.stage }
                const boundCounter = requestCount.bind(labels)
                boundInstruments.set(req.path, boundCounter)
            }
    
            boundInstruments.get(req.path).add(1)
            next()
        }
    }

    along with my handler.js

    'use strict';
    
    const { countAllRequest } = require('./counter')
    
    module.exports.hello = async (event) => {
      console.log('Calling counter')
      countAllRequest()
      console.log('Counter called')
      return {
        statusCode: 200,
        body: JSON.stringify(
          {
            message: 'Go Serverless v1.0! Your function executed successfully!',
            input: event,
          },
          null,
          2
        ),
      };
    
      // Use this code if you don't use the http event with the LAMBDA-PROXY integration
      // return { message: 'Go Serverless v1.0! Your function executed successfully!', event };
    };

    however, I don't see anything in my x-ray console. Is there something that I am doing wrong? Here's the repo: https://github.com/mfolivas/aws-serverless-nodejs-opentelemetry

    Adam Finzel
    @afinzel_twitter
    Hey, If I am geting exporting failed are there any logs to explain why it failed/
    1 reply
    Marcelo Olivas
    @mfolivas

    I don't see any metrics. I was able to get my lambda with the lambda layers for the aoc-extension: https://github.com/aws-observability/aws-otel-lambda/tree/main/extensions/aoc-extension. I configured my lambda as followed:

    
    
    service: aws-serverless-nodejs-opentelemetry
    
    frameworkVersion: '2'
    
    plugins:
      - serverless-pseudo-parameters
    
    provider:
      name: aws
      runtime: nodejs12.x
      lambdaHashingVersion: 20201221
    
      stage: dev
      tracing: true # enable tracing
      iamRoleStatements:
        - Effect: "Allow" # xray permissions (required)
          Action:
            - "xray:PutTraceSegments"
            - "xray:PutTelemetryRecords"
          Resource:
            - "*"
    
    functions:
      hello:
        handler: src/handler.hello
        layers:
          - arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:layer:aws-otel-collector:1

    and have the following handler.js with the metric-emitter.js

    'use strict';
    
    const { DiagConsoleLogger, DiagLogLevel, diag } = require('@opentelemetry/api')
    const { CollectorMetricExporter } = require('@opentelemetry/exporter-collector-grpc');
    const { MeterProvider } = require('@opentelemetry/metrics');
    
    const API_COUNTER_METRIC = 'apiBytesSent';
    const API_LATENCY_METRIC = 'latency';
    
    // Optional and only needed to see the internal diagnostic logging (during development)
    diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);
    
    /** The OTLP Metrics gRPC Collector */
    const metricExporter = new CollectorMetricExporter({
        serviceName: process.env.OTEL_RESOURCE_ATTRIBUTES ? process.env.OTEL_RESOURCE_ATTRIBUTES : 'aws-otel-integ-test',
        url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ? process.env.OTEL_EXPORTER_OTLP_ENDPOINT : 'localhost:55680'
    });
    
    /** The OTLP Metrics Provider with OTLP gRPC Metric Exporter and Metrics collection Interval  */
    const meter = new MeterProvider({
        exporter: metricExporter,
        interval: 1000,
    }).getMeter('aws-otel');
    
    /**  grabs instanceId and append to metric name to check individual metric for integration test */
    const latencyMetricName = API_LATENCY_METRIC;
    const  apiBytesSentMetricName = API_COUNTER_METRIC;
    // const instanceId = process.env.INSTANCE_ID || '';
    // if (instanceId && instanceId.trim() !== '') {
    //     latencyMetricName += '_' + instanceId;
    //     apiBytesSentMetricName += '_' + instanceId;
    // }
    
    /** Counter Metrics */
    const payloadMetric = meter.createCounter(apiBytesSentMetricName, {
        description: 'Metric for counting request payload size',
    });
    
    /** Value Recorder Metrics with Histogram */
    const requestLatency = meter.createValueRecorder(latencyMetricName, {
        description: 'Metric for record request latency'
    });
    
    //** binds request latency metric with returnTime */
    const emitReturnTimeMetric = (returnTime, apiName, statusCode) => {
        console.log('emit metric with return time ' + returnTime + ', ' + apiName + ', ' + statusCode);
        const labels = { 'apiName': apiName, 'statusCode': statusCode };
        requestLatency.bind(labels).record(returnTime);
    }
    
    //** emitsPayLoadMetrics() Binds payload Metric with number of bytes */
    const emitsPayloadMetric = (bytes, apiName, statusCode) => {
        console.log('emit metric with http request size ' + bytes + ' byte, ' + apiName);
        const labels = { 'apiName': apiName, 'statusCode': statusCode };
        payloadMetric.bind(labels).add(bytes);
    }
    
    module.exports = {
        emitReturnTimeMetric: emitReturnTimeMetric,
        emitsPayloadMetric: emitsPayloadMetric
    }

    the handler.js:

    module.exports.hello = async (event) => {
      const requestStartTime = new Date().getMilliseconds();
      //TODO fix error - Cannot read property 'context' of undefined
      // const traceID = returnTraceIdJson()
      // console.log('tracerId', traceID)
      http.get('http://aws.amazon.com', () => {
        console.log('Calling counter')
        const statusCode = 200
        meter.emitsPayloadMetric(mimicPayLoadSize(), '/outgoing-http-call', statusCode);
        meter.emitReturnTimeMetric(new Date().getMilliseconds() - requestStartTime, '/outgoing-http-call', statusCode);
        console.log('Counter called')
      })
      return {
        statusCode: 200,
        body: JSON.stringify(
          {
    1 reply
    Marcelo Olivas
    @mfolivas
    Do you by when would you expect to have the accumulator working for the nodejs? I have tried everything and it's not working.
    Also, would it contain all metrics that we currently have for lamdba insights?
    Niraj Gupta
    @sammy22

    I ran the aws-Otel-collector image locally.
    docker run --rm -p 4317:4317 -p 55680:55680 -p 8889:8888 -e AWS_REGION=us-west-2 -v ${PWD}/config.yaml:/otel-local-config.yaml -v ~/.aws:/root/.aws --name awscollector public.ecr.aws/aws-observability/aws-otel-collector:v0.8.0 --config otel-local-config.yaml;

    when I just stop the container
    I get the error
    2021-03-30T07:11:05.119Z INFO service/service.go:469 Shutdown complete.
    Error: failed to shutdown pipelines: failed to shutdown exporters: sync /dev/stderr: invalid argument

    6 replies
    Zaheer Shaik
    @zaheershaik
    Hi. What is the preferred AWS OTel collector image to use ? Is it amazon/aws-otel-collector or public.ecr.aws/aws-observability/aws-otel-collector ? The former one didn’t quite work.
    Min Xia
    @mxiamxia
    Both amazon/aws-otel-collector and public.ecr.aws/aws-observability/aws-otel-collector should work the same. But due to the new pull limitation from DockerHub. We suggest everyone to use public.ecr.aws/aws-observability/aws-otel-collector
    Zaheer Shaik
    @zaheershaik
    24 replies
    Zaheer Shaik
    @zaheershaik
    Hi. Is there a trace processor being developed which can filter traces or spans based on a pattern? Something like this? https://github.com/SumoLogic/opentelemetry-collector-contrib/tree/main/processor/cascadingfilterprocessor#policy-configuration
    2 replies
    Zaheer Shaik
    @zaheershaik
    Hi. I’ve deployed AWS OTel collector gateway in auto-scaling ECS cluster on Fargate. I’m planning to use “Tail Sampling Processor” to sample traces. I see this note in the processor repo “Technically, trace ID aware load balancing could be used to support multiple collector instances, but this configuration has not been tested.”. Could you advice on how to setup “trace ID aware load balancing” so that spans that belong to a single trace id land onto the same collector instance?
    12 replies
    Rich
    @jychen7

    hi, do we have an end to end test case for "jaeger receiver -> xray exporter"?

    I am currently using nginx-opentracing to jaeger receiver, with the debian package from CI of this PR: aws-observability/aws-otel-collector#423

    but got this error from xray exporter, "Error translating span". And I have test "nginx opentracing -> jaeger receiver -> logging export" can successfully show the spans

    (sorry, cc @mxiamxia )

    4 replies
    Zaheer Shaik
    @zaheershaik
    Hi. The documentation for signalfxexporter says that it can only receive metadata updates from k8s_cluster receiver. I don’t see the receiver support in aws-otel-collector but I see the support for signalfxexporter. Could you pls share an example of if this exporter works with otlp metrics receiver?
    https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/signalfxexporter
    Min Xia
    @mxiamxia

    Hi. The documentation for signalfxexporter says that it can only receive metadata updates from k8s_cluster receiver. I don’t see the receiver support in aws-otel-collector but I see the support for signalfxexporter. Could you pls share an example of if this exporter works with otlp metrics receiver?
    https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/signalfxexporter

    Have you tried the documentations here? - https://aws-otel.github.io/docs/partners/splunk#signalfx-exporter

    Nicolai Wendt
    @statusrapport_twitter
    Hi. I have set up ADOT with Lambda but am struggling with extending this to AWS Batch (ECS). I'm thinking of two approaches, but not sure if any of them are viable (a third option if you include using the latest cloud watch agent which should contain adot). I'd rather not pursue the first option of configuring all our jobs as multi-node parallel jobs communicating with a collector. Rather I'll keep attempting to deploy it as a deamon with every cluster (as that at least is supported for x-ray on aws batch). What approach would you recommend?