To the person that fixed the open-telemetry/opentelemetry-collector#2121 this fixed my issue with vanished metrics and dirty workaround to continously restart OTEL Collectors! thnx @kohrapha
2 replies
Could I get a quick overview of the changes that were made here? https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/1489/files I think due to some timing issues the vendor exporter i'm maintaining got missed in this PR, and starting to see some bug reports now (open-telemetry/opentelemetry-collector-contrib#1684). Is the idea that, for vendor exporters, instead of attempting to match statusCode against a range (400/500 errors), you just see if
status.Code() == pdata.StatusCodeError? I'd be happy to update the datadog exporter to match whatever the current spec is
1 reply
Hi all, the hostmetrics receiver from the OpenTelemetry Collector reports metrics namespaced under
system.swap. (see here) that are similar in meaning but different in naming to the system.paging. metrics that feature in the spec (see here). Is this expected? If not, what should be changed (spec/implementation)? Should I open an issue for this?
1 reply
Hi All, In custom processors/receivers/exporters is there a provision to send custom app metrics via the stats or obsrreport package or any other means?
For ex> in my custom processor, there is a cache of metadata which gets refreshed periodically.. if there is any problem reloading the cache I would like to emit a metric, so that i can set up the necessary monitoring..
For ex> in my custom processor, there is a cache of metadata which gets refreshed periodically.. if there is any problem reloading the cache I would like to emit a metric, so that i can set up the necessary monitoring..
@tannaga yes, see this example: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/master/processor/groupbytraceprocessor/metrics.go
1 reply
make otelcol;
go generate ./...
receiver/hostmetricsreceiver/codegen.go:17: running "mdatagen": exec: "mdatagen": executable file not found in $PATH
service/internal/gen.go:18: running "addlicense": exec: "addlicense": executable file not found in $PATH
make: * [otelcol] Error 1
go generate ./...
receiver/hostmetricsreceiver/codegen.go:17: running "mdatagen": exec: "mdatagen": executable file not found in $PATH
service/internal/gen.go:18: running "addlicense": exec: "addlicense": executable file not found in $PATH
make: * [otelcol] Error 1
I tried sunning ''go get -u github.com/mjibson/esc'' making sure $GOPATH and PATH are set
I installed go via homebrew
This is how my zshrc looks like - cat ~/.zshrc
export GOPATH=/usr/local/Cellar/go/1.15.5
export PATH=$PATH:$(go env GOPATH)/bin
export GOPATH=/usr/local/Cellar/go/1.15.5
export PATH=$PATH:$(go env GOPATH)/bin
I opened an issue in opentelemetry-collector core that is a continuation of a discussion i was having in contrib, it involves a number of vendors, clients, and formats, and input from maintainers is appreciated open-telemetry/opentelemetry-collector#2214 . I will try to attend mtg today to sync on this as well. cheers
cross-posting from opentelemetry-specification
Friendly reminder we have
otel spec issue scrub 🧽mtg tomorrow, Friday morning 8:30a PST.
Not many spec issues to triage, so if we have quorum with collector maintainers, we’ll also continue triaging collector and collector-contrib issues.
Hi all, would love a review on these PRs:
@bogdandrutu you might have missed the notifications on github, so, wanted to ping you here about this: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/1736#discussion_r534010815
@bogdandrutu (pinging you because you worked on the FluentBit loadtests)
I've been working on migrating the CircleCI workflows to GitHub Actions and there is just 1 job left to do; loadtest. In loadtest all the tests are passing except TestLog10kDPS/FluentBitToOTLPwhich I think might have an error in the implementation. I'll attach the logs of the run in thread but the main issue im running into is that im getting conflicting messages from the logs on whether the FluentBItWriter can connect to the NewOTLPDataReceiver or not. On one hand, the logs are filled with a whole bunch of Sent: X items| Received: 0)
2020/12/15 00:07:48 Agent RAM (RES): 49 MiB, CPU: 1.3% | Sent: 59600 items | Received: 0 items (0/sec)
2020/12/15 00:07:51 Agent RAM (RES): 49 MiB, CPU: 0.3% | Sent: 89600 items | Received: 0 items (0/sec)
2020/12/15 00:07:54 Agent RAM (RES): 49 MiB, CPU: 0.0% | Sent: 119600 items | Received: 0 items (0/sec)
2020/12/15 00:07:57 Agent RAM (RES): 49 MiB, CPU: 0.0% | Sent: 149600 items | Received: 0 items (0/sec)
...
validator.go:46:
Error Trace: validator.go:46
test_case.go:273
scenarios.go:190
log_test.go:78
Error: Not equal:
expected: 0x249f0
actual : 0x0
Test: TestLog10kDPS/FluentBitToOTLP
Messages: Received and sent counters do not match.but then I also get the following error message:
test_case.go:312: Time out waiting for [all data items received]This is weird because test_case.go:312 is tc.t.Error("Time out waiting for", errMsg) meaning the errMsg in the log is saying all data items were received (which is isn't really an error 🤔).
Every other writer for every other test is able to connect to the NewOTLPDataReceiverthough but I dont know enough about the fluentBit exporter to know why this fails.
Any help would be appreciated, thanks!
6 replies
ah nvm, turns out the issue was this over here: https://github.com/open-telemetry/opentelemetry-collector/blob/master/testbed/testbed/receivers.go#L63. Works with 127.0.0.1 instead
hi - I thought I would start testing hostmetrics as a first opentelemetry test, but it seems like exporting metrics to Azure Monitor / Application Insights isn't supported (in -contrib). Is this planned? Are Microsoft planning to officially support an exporter, like many other commercial product owners?
@/all I will be on PTO starting from tomorrow until Jan 4. I will attend today's SIG meeting and then will be out for the next 2 meetings. I am not sure what the attendance is going to look like given the holidays, but I will keep the meetings on the calendar in case there is people who want to meet.
Hello, I'm a bit confused about queued_retry, I bumped opentelemetry-collector to 0.17.0 and as I had queued_retry processor specified, it produced WARNs and agents was unable to perform that retry (I know - it's deprecated). I know I should use exporter helper, but is it enabled by default? (so it should be enough to just remove queued_retry from processor, and the functionality of those retries would remain with version 0.17.0?)
6 replies
Hi @bogdandrutu can you have a second look for this "filterprocessor" PR open-telemetry/opentelemetry-collector#2251
@bogdandrutu thanks for merging PR 1/3! I've rebased and pushed PR 2/3 open-telemetry/opentelemetry-collector#2298 :smile:
Friends, is there a processor that permits span attribute => resource attribute mapping? A number of use cases here but sometimes a non otel client will add
host info as a span attribute that we'd like to mapped to the appropriate resource attribute key so that various exporters can pick it up correctly as the hostname info
Is it true that when
queued_retry processor is doing retry on failure, and if it continues failing, it will be trying to retry it in endless loop till it will successfully retry? (so there is no upper bound that tells it to stop when it failed to retry too many times, or for some period of time)
1 reply
Hi, I'm trying to get a PR green on
collector-contrib. But I seem to get Lint errors on code that isn't related to my PR. Can someone give me some hints? open-telemetry/opentelemetry-collector-contrib#1892
7 replies
hi friends, have a PR that's been approved by a number of folks, would be great to see this merged before next release as it's been 9 days now open-telemetry/opentelemetry-collector#2253
And two others on contrib have been approved and sitting for a little while that i'd been keen to see merged before next release
- open-telemetry/opentelemetry-collector-contrib#1895
- open-telemetry/opentelemetry-collector-contrib#1861
these are all impacting end users so would be a great help for adoption, cheers!
thought this might be interesting once I saw it was from somoene at datadog and worked on ddsketch implementation https://richardstartin.github.io/posts/dont-use-protobuf-for-telemetry
1 reply
Hello Friends, I have submitted a PR open-telemetry/opentelemetry-collector-contrib#1887 to collector contrib couple of weeks back - it is currently approved state - any estimate on when it can be expected in master. A kind request to include this change in the upcoming release - Please and Thank you
Hi All, in the last Collector SIG, there was an issue raised about how best to handle pipelines where it may be necessary to translate from one signal type to another. I drafted a proposal enhancement to address this type of scenario and would appreciate any thoughts the community may have on it: open-telemetry/opentelemetry-collector#2336
Hi all, I submitted this PR about a week ago and would appreciate if somebody could please take a look at it? It's mostly config, docs, tests and skeleton code.
open-telemetry/opentelemetry-collector-contrib#1917
open-telemetry/opentelemetry-collector-contrib#1917
Hi @bogdandrutu would be great if you get time to have a look on this PR- open-telemetry/opentelemetry-collector#2251
Hi, we would like to get this PR merged before 0.18.0 is released, it is a small change on a previous PR by me open-telemetry/opentelemetry-collector-contrib#1962 It has been already reviewed by someone from Datadog. Could it be merged?
Hi,
I are using otel collector and with open census and we are getting this error
exporterhelper/queued_retry.go:239 Exporting failed. The error is not retryable. Dropping data. {"component_kind": "exporter", "component_type": "prometheusremotewrite", "component_name": "prometheusremotewrite", "error": "Permanent error: [Permanent error: nil data point. opencensus.io/http/client/roundtrip_latency is dropped; Permanent error: nil data point. opencensus.io/http/client/received_bytes is dropped; Permanent error: nil data point. grpc.io/server/sent_messages_per_rpc is dropped; Permanent error: nil data point. grpc.io/server/received_bytes_per_rpc is dropped; Permanent error: nil data point. grpc.io/server/sent_bytes_per_rpc is dropped; Permanent error: nil data point. opencensus.io/http/client/sent_bytes is dropped; Permanent error: nil data point. grpc.io/server/received_messages_per_rpc is dropped; Permanent error: nil data point. grpc.io/server/server_latency is dropped; Permanent error: nil data point. grpc.io/server/completed_rpcs is dropped]", "dropped_items": 9}screenshot
the config is :
extensions:
health_check:
pprof:
endpoint: 0.0.0.0:1777
receivers:
opencensus:
endpoint: "0.0.0.0:55678"
processors:
batch:
memory_limiter:
ballast_size_mib: 683
limit_mib: 1500
spike_limit_mib: 512
check_interval: 5s
queued_retry:
exporters:
logging:
logLevel: debug
jaeger:
endpoint: <jaeger_collector_url>
insecure: true
prometheusremotewrite:
endpoint: <prom_remote_url>
service:
pipelines:
traces:
receivers: [opencensus]
processors: [memory_limiter, batch, queued_retry]
exporters: [jaeger]
metrics:
receivers: [opencensus]
processors: [memory_limiter, batch, queued_retry]
exporters: [prometheusremotewrite]
extensions: [health_check, pprof]Please help