Thanks @schofield, we'll see what's going on with MSK. I've opened timberio/vector#1833 to look into it.
Regarding the rate limit error, you just need to raise https://vector.dev/docs/reference/sinks/humio_logs/#rate_limit_num . Let me know if we should raise the default there.
Hello, I am routing glog file to elasticsearch.
[transforms.add_metadata]
type = "lua"
inputs = ["src"]
source = """
event['app'] = event['file']:match('/log/(.+)%.INFO')
"""
[sinks.proj_sink]
type = "elasticsearch"
inputs = ["add_metadata"]
host = "http://1.2.3.4:5"
index = "proj-{{host}}-{{app}}"
The above snippet does not work. The log shows nothing.
However
index = "proj-{{host}}"
works and msgs have correct app
field. Why can't I use {{app}} in index?
Hi all, I'm not able to get the aws_ec2_metadata
transform working. It seems to be connecting correctly to the metadata endpoint, but never makes a request. Here's my sample config:
data_dir = "/var/lib/vector"
[sources.stdin]
type = "stdin"
[transforms.add_aws_metadata]
type = "aws_ec2_metadata"
inputs = ["stdin"]
[sinks.console]
type = "console"
inputs = ["add_aws_metadata"]
encoding = "json"
And the debug logs:
Feb 19 21:59:30.069 INFO vector: Log level "info" is enabled.
Feb 19 21:59:30.069 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Feb 19 21:59:30.072 INFO vector: Vector is starting. version="0.8.0" git_version="v0.7.0-168-g841a8f8" released="Wed, 19 Feb 2020 11:14:46 +0000" arch="x86_64"
Feb 19 21:59:30.073 DEBUG trust_dns_resolver::async_resolver::background: trust-dns resolver running
Feb 19 21:59:30.073 INFO vector::topology: Running healthchecks.
Feb 19 21:59:30.073 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 21:59:30.074 INFO vector::topology: Starting source "stdin"
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))
Feb 19 21:59:30.075 INFO vector::topology::builder: Healthcheck: Passed.
Feb 19 21:59:30.076 INFO vector::topology: Starting transform "add_aws_metadata"
Feb 19 21:59:30.076 INFO source{name=stdin type=stdin}: vector::sources::stdin: Capturing STDIN
Feb 19 21:59:30.076 INFO vector::topology: Starting sink "console"
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 4194305
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 1
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 8388610
Feb 19 21:59:30.076 DEBUG tokio_reactor::background: starting background reactor
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 2
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 12582915
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 3
Feb 19 21:59:30.077 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG hyper::proto::h1::io: flushed 102 bytes
test
{"host":"6792b4ce6d87","timestamp":"2020-02-19T21:59:37.278494604Z","message":"test"}
I'm running inside Docker using the nightly-alpine image. Any help/thoughts appreciated!
Actually, if I wait long enough I see
Feb 19 22:02:17.835 DEBUG hyper::proto::h1::conn: parse error (connection error: Connection reset by peer (os error 104)) with 0 bytes
Feb 19 22:02:17.835 DEBUG hyper::proto::h1::dispatch: read_head error: connection error: Connection reset by peer (os error 104)
Feb 19 22:02:17.835 DEBUG tokio_reactor: dropping I/O source: 0
Feb 19 22:02:17.836 ERROR aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Unable to fetch EC2 metadata; Retrying. error=connection error: Connection reset by peer (os error 104)
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))
Maybe there's something else going on - although I'm able to curl the metadata endpoint from inside the container...
admin@ip-172-20-98-28:~$ sudo docker run -it --entrypoint /bin/sh -e LOG=debug --rm --name vector -v $PWD/vector.toml:/etc/vector/vector.toml -v /var/lib/docker:/var/lib/docker -v /var/run/docker.sock:/var/run/docker.sock -v
$PWD/vector:/var/lib/vector -v /var/log/pods:/var/log/pods timberio/vector:nightly-alpine
/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/3) Installing nghttp2-libs (1.39.2-r0)
(2/3) Installing libcurl (7.66.0-r0)
(3/3) Installing curl (7.66.0-r0)
Executing busybox-1.30.1-r3.trigger
OK: 10 MiB in 19 packages
/ # curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
"accountId" : "------------",
"architecture" : "x86_64",
"availabilityZone" : "ca-central-1a",
"billingProducts" : null,
"devpayProductCodes" : null,
"marketplaceProductCodes" : null,
"imageId" : "ami-0xxxxxx",
"instanceId" : "i-0xxxxxx",
"instanceType" : "m5.xlarge",
"kernelId" : null,
"pendingTime" : "2020-02-11T15:42:59Z",
"privateIp" : "172.20.98.28",
"ramdiskId" : null,
"region" : "ca-central-1",
"version" : "2017-09-30"
}/
--net=host
, but also it seems that it's only the /latest/api/token
endpoint that is timing out from within a container. It seems that the API that should be used instead is the IMDS metadata - botocore updated due to this issue and you can see the changes here: boto/botocore#1895
drop_field
and field
but it didn't workHello someone has try AWS S3 Sink with ceph ? For me it doesn't work, for example for the healtcheck ceph return a 404 response code for the head method while it return 200 response code when i'm using mc ls, here is the config
[sinks.ceph]
# REQUIRED - General
type = "aws_s3" # must be: "aws_s3"
inputs = ["syslog"] # example
bucket = "vector" # example
compression = "none" # example, enum
endpoint = "http://my-ceph.com:9000"
# OPTIONAL - Object Names
filename_append_uuid = true # default
filename_extension = "log" # default
filename_time_format = "%s" # default
key_prefix = "date=%F/" # default
# REQUIRED - requests
encoding = "text" # example, enum
# OPTIONAL - General
healthcheck = true# default
i set also the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY . When i try to send a log it returns meFeb 28 16:40:05.185 ERROR sink{name=ceph type=aws_s3}: vector::sinks::util::retries: encountered non-retriable error. error=<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>http://my-ceph.com:9000</BucketName><RequestId>tx00000000000000c51a948-005e594265-430c8a-myhost-1</RequestId><HostId>myhostid</HostId></Error>
Feb 28 16:40:05.185 ERROR sink{name=ceph type=aws_s3}: vector::sinks::util: request failed. error=<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>http://my-ceph.com:9000</BucketName><RequestId>tx00000000000000c51a948-005e594265-430c8a-myhost-1</RequestId><HostId>myhostid</HostId></Error>
Could you help me with that please ? :-) Have a nice day
hello
https://github.com/prometheus/statsd_exporter
Note that timers will be accepted with the ms, h, and d statsd types. The first two are timers and histograms and the d type is for DataDog's "distribution" type. The distribution type is treated identically to timers and histograms.
does vector support DD type? do we need to create issue?
Hey - I have a small problem with reloading configurations. If the source is http / logplex / splunk_hec (all of which use Warp) and you change the configuration, but don't change the port, I get a configuration error (address already in use) and the reload fails. Workaround is to just change the port to a new value. After a successful reload you can then change the port back to the original.
It's not a huge issue, but I wanted to see if it was known.
ERROR vector::topology: Configuration error: Source "in": Address already in use (os error 48)
ERROR vector: Reload was not successful.
Hi, guys! Small question regarding config for kafka sink - I have next piece of config
[sinks.kafka]
type = "kafka"
inputs = ["json"]
bootstrap_servers = "kafka-server:9092"
topic = "vector"
compression = "none"
healthcheck = true
buffer.type = "disk"
buffer.max_size = 104900000
buffer.when_full = "block"
encoding.codec = "json"
And when I try to start vector I get:
unknown variant `codec`, expected `text` or `json` for key `sinks.kafka`
What's wrong with config?
Thanks!