Hi All,
I am finishing implementation of influxdb sink
and I have one question about integration tests. Where is a correct place to start InfluxDB for integration tests?
It’s ok if i add influxdb-docker image into docker-compose.yml and integration tests from influxdb_metrics.rs
use it?
You can check progress of influxdb sink
here: https://github.com/bonitoo-io/vector/blob/influxdb_metrics/src/sinks/influxdb_metrics.rs
Regards
cc @loony-bean @lukesteensen
hello everyone
I have an issue with connecting to kafka from docker
current config:
[sources.kafka]
type = "kafka"
bootstrap_servers = "absolutly-sure-no-such-kafka-host"
group_id = "elb-logs-consumer"
topics = ["elb-logs"]
[sinks.out]
inputs = ["kafka"]
type = "console"
encoding = "text"
args:
Args:
--config
/config/vector.toml
--require-healthy
--verbose
looks like healtheck does not work:
Feb 07 13:34:42.017 INFO vector: Log level "debug" is enabled.
Feb 07 13:34:42.017 INFO vector: Loading config. path="/config/vector.toml"
Feb 07 13:34:42.018 INFO vector: Vector is starting. version="0.7.2" git_version="v0.7.2" released="Fri, 31 Jan 2020 20:33:36 +0000" arch="x86_64"
Feb 07 13:34:42.019 INFO vector::topology: Running healthchecks.
Feb 07 13:34:42.019 INFO vector::topology::builder: Healthcheck: Passed.
Feb 07 13:34:42.020 INFO vector::topology: All healthchecks passed.
Feb 07 13:34:42.020 INFO vector::topology: Starting source "kafka"
Feb 07 13:34:42.020 INFO vector::topology: Starting sink "out"
did I miss something?
[sources.kafka]
type = "kafka" # must be: "kafka"
bootstrap_servers = "broker1:9092,broker2:9092,broker3:9092"
group_id = "consumer-group" # example
topics = ["test-logs"] # example
Thanks @schofield, we'll see what's going on with MSK. I've opened timberio/vector#1833 to look into it.
Regarding the rate limit error, you just need to raise https://vector.dev/docs/reference/sinks/humio_logs/#rate_limit_num . Let me know if we should raise the default there.
Hello, I am routing glog file to elasticsearch.
[transforms.add_metadata]
type = "lua"
inputs = ["src"]
source = """
event['app'] = event['file']:match('/log/(.+)%.INFO')
"""
[sinks.proj_sink]
type = "elasticsearch"
inputs = ["add_metadata"]
host = "http://1.2.3.4:5"
index = "proj-{{host}}-{{app}}"
The above snippet does not work. The log shows nothing.
However
index = "proj-{{host}}"
works and msgs have correct app
field. Why can't I use {{app}} in index?
Hi all, I'm not able to get the aws_ec2_metadata
transform working. It seems to be connecting correctly to the metadata endpoint, but never makes a request. Here's my sample config:
data_dir = "/var/lib/vector"
[sources.stdin]
type = "stdin"
[transforms.add_aws_metadata]
type = "aws_ec2_metadata"
inputs = ["stdin"]
[sinks.console]
type = "console"
inputs = ["add_aws_metadata"]
encoding = "json"
And the debug logs:
Feb 19 21:59:30.069 INFO vector: Log level "info" is enabled.
Feb 19 21:59:30.069 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Feb 19 21:59:30.072 INFO vector: Vector is starting. version="0.8.0" git_version="v0.7.0-168-g841a8f8" released="Wed, 19 Feb 2020 11:14:46 +0000" arch="x86_64"
Feb 19 21:59:30.073 DEBUG trust_dns_resolver::async_resolver::background: trust-dns resolver running
Feb 19 21:59:30.073 INFO vector::topology: Running healthchecks.
Feb 19 21:59:30.073 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 21:59:30.074 INFO vector::topology: Starting source "stdin"
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))
Feb 19 21:59:30.075 INFO vector::topology::builder: Healthcheck: Passed.
Feb 19 21:59:30.076 INFO vector::topology: Starting transform "add_aws_metadata"
Feb 19 21:59:30.076 INFO source{name=stdin type=stdin}: vector::sources::stdin: Capturing STDIN
Feb 19 21:59:30.076 INFO vector::topology: Starting sink "console"
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 4194305
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 1
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 8388610
Feb 19 21:59:30.076 DEBUG tokio_reactor::background: starting background reactor
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 2
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 12582915
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 3
Feb 19 21:59:30.077 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG hyper::proto::h1::io: flushed 102 bytes
test
{"host":"6792b4ce6d87","timestamp":"2020-02-19T21:59:37.278494604Z","message":"test"}
I'm running inside Docker using the nightly-alpine image. Any help/thoughts appreciated!
Actually, if I wait long enough I see
Feb 19 22:02:17.835 DEBUG hyper::proto::h1::conn: parse error (connection error: Connection reset by peer (os error 104)) with 0 bytes
Feb 19 22:02:17.835 DEBUG hyper::proto::h1::dispatch: read_head error: connection error: Connection reset by peer (os error 104)
Feb 19 22:02:17.835 DEBUG tokio_reactor: dropping I/O source: 0
Feb 19 22:02:17.836 ERROR aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Unable to fetch EC2 metadata; Retrying. error=connection error: Connection reset by peer (os error 104)
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))
Maybe there's something else going on - although I'm able to curl the metadata endpoint from inside the container...
admin@ip-172-20-98-28:~$ sudo docker run -it --entrypoint /bin/sh -e LOG=debug --rm --name vector -v $PWD/vector.toml:/etc/vector/vector.toml -v /var/lib/docker:/var/lib/docker -v /var/run/docker.sock:/var/run/docker.sock -v
$PWD/vector:/var/lib/vector -v /var/log/pods:/var/log/pods timberio/vector:nightly-alpine
/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/3) Installing nghttp2-libs (1.39.2-r0)
(2/3) Installing libcurl (7.66.0-r0)
(3/3) Installing curl (7.66.0-r0)
Executing busybox-1.30.1-r3.trigger
OK: 10 MiB in 19 packages
/ # curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
"accountId" : "------------",
"architecture" : "x86_64",
"availabilityZone" : "ca-central-1a",
"billingProducts" : null,
"devpayProductCodes" : null,
"marketplaceProductCodes" : null,
"imageId" : "ami-0xxxxxx",
"instanceId" : "i-0xxxxxx",
"instanceType" : "m5.xlarge",
"kernelId" : null,
"pendingTime" : "2020-02-11T15:42:59Z",
"privateIp" : "172.20.98.28",
"ramdiskId" : null,
"region" : "ca-central-1",
"version" : "2017-09-30"
}/
--net=host
, but also it seems that it's only the /latest/api/token
endpoint that is timing out from within a container. It seems that the API that should be used instead is the IMDS metadata - botocore updated due to this issue and you can see the changes here: boto/botocore#1895