vector/community

https://vector.dev/

Binary Logic

@binarylogic

We haven’t tested explicitly with Amazon’s Kafka service, so it might be that, but my assumption is that it shouldn’t matter.

Grant Schofield

@schofield

cool, also I got a strange error regarding humio, there aren't a lot of logs I'm shipping when using the file source but see Feb 15 01:16:26.584 TRACE sink{name=humio type=humio_logs}: tower_limit::rate::service: rate limit exceeded, disabling service

will check back Monday

Oh and the Kafka did work fine with our own kafka connect sink, and kafkacat works...

Binary Logic

@binarylogic

Thanks @schofield, we'll see what's going on with MSK. I've opened timberio/vector#1833 to look into it.

Regarding the rate limit error, you just need to raise https://vector.dev/docs/reference/sinks/humio_logs/#rate_limit_num . Let me know if we should raise the default there.

Grant Schofield

@schofield

thanks, missed that one, will give it a go

Andrey Afoninsky

@afoninsky

just noticed:
you've added "loki" sync as a part of the sprint
and it appeared in documentation which is good: https://vector.dev/docs/reference/sinks/loki/
the problem is - there is no information that it's a part of a nightly build, and not a part of latest stable (0.7.2) so it does not work

5 replies

leidruid

@leidruid_gitlab

hello, guys! I use vector as tcp > kafka transport.
some short time after the start, sink stucks, nothing happens, no suspicious messages in broker and vector logs (in debug mode too)

for a litte time before stuck, in vector log disappears sink messages (only source messages remains)

Has anyone encountered this problem?

4 replies

elbaro

@elbaro

Hello, I am routing glog file to elasticsearch.

[transforms.add_metadata] 
  type = "lua"
  inputs = ["src"]
  source = """
event['app'] = event['file']:match('/log/(.+)%.INFO')
"""

[sinks.proj_sink]
  type = "elasticsearch"
  inputs = ["add_metadata"]
  host = "http://1.2.3.4:5"
  index = "proj-{{host}}-{{app}}"

The above snippet does not work. The log shows nothing.

However

  index = "proj-{{host}}"

works and msgs have correct app field. Why can't I use {{app}} in index?

2 replies

Samuel Cormier-Iijima

@sciyoshi

Hi all, I'm not able to get the aws_ec2_metadata transform working. It seems to be connecting correctly to the metadata endpoint, but never makes a request. Here's my sample config:

data_dir = "/var/lib/vector"

[sources.stdin]
  type = "stdin"

[transforms.add_aws_metadata]
  type = "aws_ec2_metadata"
  inputs = ["stdin"]

[sinks.console]
  type = "console"
  inputs = ["add_aws_metadata"]
  encoding = "json"

And the debug logs:

Feb 19 21:59:30.069  INFO vector: Log level "info" is enabled.
Feb 19 21:59:30.069  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Feb 19 21:59:30.072  INFO vector: Vector is starting. version="0.8.0" git_version="v0.7.0-168-g841a8f8" released="Wed, 19 Feb 2020 11:14:46 +0000" arch="x86_64"
Feb 19 21:59:30.073 DEBUG trust_dns_resolver::async_resolver::background: trust-dns resolver running
Feb 19 21:59:30.073  INFO vector::topology: Running healthchecks.
Feb 19 21:59:30.073 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 21:59:30.074  INFO vector::topology: Starting source "stdin"
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))
Feb 19 21:59:30.075  INFO vector::topology::builder: Healthcheck: Passed.
Feb 19 21:59:30.076  INFO vector::topology: Starting transform "add_aws_metadata"
Feb 19 21:59:30.076  INFO source{name=stdin type=stdin}: vector::sources::stdin: Capturing STDIN
Feb 19 21:59:30.076  INFO vector::topology: Starting sink "console"
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 4194305
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 1
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 8388610
Feb 19 21:59:30.076 DEBUG tokio_reactor::background: starting background reactor
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 2
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 12582915
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 3
Feb 19 21:59:30.077 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG hyper::proto::h1::io: flushed 102 bytes
test
{"host":"6792b4ce6d87","timestamp":"2020-02-19T21:59:37.278494604Z","message":"test"}

I'm running inside Docker using the nightly-alpine image. Any help/thoughts appreciated!

Actually, if I wait long enough I see

Feb 19 22:02:17.835 DEBUG hyper::proto::h1::conn: parse error (connection error: Connection reset by peer (os error 104)) with 0 bytes
Feb 19 22:02:17.835 DEBUG hyper::proto::h1::dispatch: read_head error: connection error: Connection reset by peer (os error 104)
Feb 19 22:02:17.835 DEBUG tokio_reactor: dropping I/O source: 0
Feb 19 22:02:17.836 ERROR aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Unable to fetch EC2 metadata; Retrying. error=connection error: Connection reset by peer (os error 104)
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))

Maybe there's something else going on - although I'm able to curl the metadata endpoint from inside the container...

Lucio Franco

@LucioFranco

What type of instance are you on? and what does your curl command look like

Samuel Cormier-Iijima

@sciyoshi

thanks for the quick response @LucioFranco! it's a standard EC2 instance, m5.xlarge. here's the command I'm running:

admin@ip-172-20-98-28:~$ sudo docker run -it --entrypoint /bin/sh -e LOG=debug --rm --name vector -v $PWD/vector.toml:/etc/vector/vector.toml -v /var/lib/docker:/var/lib/docker -v /var/run/docker.sock:/var/run/docker.sock -v
$PWD/vector:/var/lib/vector -v /var/log/pods:/var/log/pods timberio/vector:nightly-alpine
/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/3) Installing nghttp2-libs (1.39.2-r0)
(2/3) Installing libcurl (7.66.0-r0)
(3/3) Installing curl (7.66.0-r0)
Executing busybox-1.30.1-r3.trigger
OK: 10 MiB in 19 packages
/ # curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "------------",
  "architecture" : "x86_64",
  "availabilityZone" : "ca-central-1a",
  "billingProducts" : null,
  "devpayProductCodes" : null,
  "marketplaceProductCodes" : null,
  "imageId" : "ami-0xxxxxx",
  "instanceId" : "i-0xxxxxx",
  "instanceType" : "m5.xlarge",
  "kernelId" : null,
  "pendingTime" : "2020-02-11T15:42:59Z",
  "privateIp" : "172.20.98.28",
  "ramdiskId" : null,
  "region" : "ca-central-1",
  "version" : "2017-09-30"
}/

Lucio Franco

@LucioFranco

ah looks like you're running vector within a container, that may be the reason

Samuel Cormier-Iijima

@sciyoshi

the curl command is also running from within the container

Lucio Franco

@LucioFranco

@sciyoshi can you try running the docker command with --net=host?

Samuel Cormier-Iijima

@sciyoshi

oh yup, that worked!! thank you :) not sure why curl would have been able to connect?

Lucio Franco

@LucioFranco

I would assume black magic :) glad that worked! let us know if you have any other issues.

Samuel Cormier-Iijima

@sciyoshi

I have another quick question - the json_parser transform seems to always remove the source field when drop_field is true. This seems inconsistent with the behavior of e.g. grok_parser, which only removes it when the parse succeeds. Is that behavior intentional?

Binary Logic

@binarylogic

Hey @sciyoshi , the behavior should be consistent across the two. I've opened timberio/vector#1861 to fix that.

Sebastian YEPES

@syepes

Small question, Is it currently possible to ingest (receive from UDP, TCP or file) metrics using the line protocol?

2 replies

Samuel Cormier-Iijima

@sciyoshi

@LucioFranco update on the original issue - I'm not able to use --net=host, but also it seems that it's only the /latest/api/token endpoint that is timing out from within a container. It seems that the API that should be used instead is the IMDS metadata - botocore updated due to this issue and you can see the changes here: boto/botocore#1895

7 replies

Andrey Afoninsky

@afoninsky

does vector have loggly support? haven't found any issues about it: https://github.com/timberio/vector/search?q=loggly&unscoped_q=loggly

1 reply

Aleksey Shirokih

@freeseacher

Hi! how can i transform something like that "file":"/var/log/mysystem/subsystem-component_name-07.log" to component_name ?

1 reply

Ana Hobden

@Hoverbear

Glad you got it!

Aleksey Shirokih

@freeseacher

as i can see there are type https://vector.dev/docs/about/data-model/metric/#aggregated_summary but how can i get it ? i am interested in prometheus summary of cause. there are some reference to timberio/vector#710 but i can't catch the point

Ana Hobden

@Hoverbear

@freeseacher if you're taking in logs and want to output metrics please try https://vector.dev/docs/reference/transforms/log_to_metric/

Aleksey Shirokih

@freeseacher

yes i am talking about metrics and already found log2metric but it does not help. type must must be one of: "counter" "gauge" "histogram" "set" but not quantile.

Samuel Cormier-Iijima

@sciyoshi

I am having issues with Docker log rotation using the default json-file logging driver - Vector stops picking up logs after the file is rotated

25 replies

Cédric Da Fonseca

@Kwelity

Hi, I'm not sure to understand how the regex transform works.
I'm trying to only parse error log message, so I have a regexp starting with "^ERROR.*", I'm expecting the transform to drop the log that doesn't match. But, the log is parsed and the log content is put in the "message" field.
I tried to play with drop_field and field but it didn't work
What would be the best solution for my use case ?

2 replies

Heinz N. Gies

@Licenser

it worked :D

Ana Hobden

@Hoverbear

Gitter: It works sometimes! :)

mlki

@MlkiTouch_twitter

Hello someone has try AWS S3 Sink with ceph ? For me it doesn't work, for example for the healtcheck ceph return a 404 response code for the head method while it return 200 response code when i'm using mc ls, here is the config

[sinks.ceph]
  # REQUIRED - General
  type = "aws_s3" # must be: "aws_s3"
  inputs = ["syslog"] # example
  bucket = "vector" # example
  compression = "none" # example, enum
  endpoint = "http://my-ceph.com:9000"

  # OPTIONAL - Object Names
  filename_append_uuid = true # default
  filename_extension = "log" # default
  filename_time_format = "%s" # default
  key_prefix = "date=%F/" # default
  # REQUIRED - requests
  encoding = "text" # example, enum

  # OPTIONAL - General
  healthcheck = true# default

i set also the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY . When i try to send a log it returns me
Feb 28 16:40:05.185 ERROR sink{name=ceph type=aws_s3}: vector::sinks::util::retries: encountered non-retriable error. error=<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>http://my-ceph.com:9000</BucketName><RequestId>tx00000000000000c51a948-005e594265-430c8a-myhost-1</RequestId><HostId>myhostid</HostId></Error> Feb 28 16:40:05.185 ERROR sink{name=ceph type=aws_s3}: vector::sinks::util: request failed. error=<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>http://my-ceph.com:9000</BucketName><RequestId>tx00000000000000c51a948-005e594265-430c8a-myhost-1</RequestId><HostId>myhostid</HostId></Error>
Could you help me with that please ? :-) Have a nice day

mahsoud

@mahsoud

Hey everyone, just started playing with vector agent on Windows to collect logs from a legacy application. In my case, when the application starts it writes a very long line into the log file (\u0000 on repeat)... what transform would you suggest to use to drop that one line?

Andrey Afoninsky

@afoninsky

hello
https://github.com/prometheus/statsd_exporter

Note that timers will be accepted with the ms, h, and d statsd types. The first two are timers and histograms and the d type is for DataDog's "distribution" type. The distribution type is treated identically to timers and histograms.

does vector support DD type? do we need to create issue?

2 replies

Andrey Afoninsky

@afoninsky

https://medium.com/@valyala/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350
does it make sense to create an issue with implementation request for prometheus sync?
pros: a better histogram (less cardinality, more accuracy)
cons: VictoriaMetric specific only, maybe it's useful in specific cases only

1 reply

ChethanU

@ChethanUK

Is there Offical helm chart?

2 replies

Bill

@bill-bateman

Hey - I have a small problem with reloading configurations. If the source is http / logplex / splunk_hec (all of which use Warp) and you change the configuration, but don't change the port, I get a configuration error (address already in use) and the reload fails. Workaround is to just change the port to a new value. After a successful reload you can then change the port back to the original.

It's not a huge issue, but I wanted to see if it was known.

ERROR vector::topology: Configuration error: Source "in": Address already in use (os error 48)
ERROR vector: Reload was not successful.

leidruid

@leidruid_gitlab

hello! Is there a correct way to specify multiple targets in elasticsearch sink, as in logstash?

9 replies

Andrey Afoninsky

@afoninsky

hello, please fix me if I wrong: "vector" source is a grpc-server and I can send logs/metrics directly using https://github.com/timberio/vector/blob/master/proto/event.proto ?

2 replies

Andrey Afoninsky

@afoninsky

another question: what's the best approach to implement log rotation / truncate in https://vector.dev/docs/reference/sinks/file/ and docker image? do you want to have an issue about it or it should be achieved using external tools? for now, I'm launching logrotate docker image as sidecar :)

2 replies

Andrey Afoninsky

@afoninsky

please take a look if you have a free time, can't understand is it a bug or my misunderstanding of documentation :) thx
timberio/vector#2036

2 replies

Andrey Afoninsky

@afoninsky

one more question :) "file sink" does not recreate a file if old one was deleted using "rm" - is it correct behaviour?

7 replies

gtie

@gtie

anyone else seeing behavior like timberio/vector#2080 ?

mmacedo

@mmacedoeu

hi, building vector using make or using docker as stated on https://vector.dev/docs/setup/installation/manual/from-source/ generates a debug mode binary

1 reply

is there any instruction I am missing to generate a release version ?

mmacedo

@mmacedoeu

I found that hotmic is deprecated, https://github.com/timberio/vector/blob/master/lib/tracing-metrics/Cargo.toml#L9 do you plan to replace it with crate metrics ?

6 replies

Serhii M.

@mikhno-s

Hi, guys! Small question regarding config for kafka sink - I have next piece of config

[sinks.kafka]
  type = "kafka"
  inputs = ["json"]
  bootstrap_servers = "kafka-server:9092"
  topic = "vector"
  compression = "none"
  healthcheck = true

  buffer.type = "disk"
  buffer.max_size = 104900000
  buffer.when_full = "block"

  encoding.codec = "json"

And when I try to start vector I get:

unknown variant `codec`, expected `text` or `json` for key `sinks.kafka`

What's wrong with config?

Thanks!

Where communities thrive

People

Repo info

Activity