by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 07:25
    perfectayush synchronize #3373
  • 07:20
    perfectayush synchronize #3373
  • 04:47
    juchiast edited #3286
  • 04:37
    juchiast synchronize #3286
  • 01:01
    lukesteensen closed #3416
  • Aug 11 22:34
    lukesteensen synchronize #3416
  • Aug 11 20:20
    bruceg closed #3094
  • Aug 11 19:58
    bruceg synchronize #3094
  • Aug 11 19:32
    lukesteensen synchronize #3416
  • Aug 11 19:22
    fanatid review_requested #3422
  • Aug 11 19:22
    fanatid review_requested #3422
  • Aug 11 19:22
    fanatid labeled #3422
  • Aug 11 19:22
    fanatid labeled #3422
  • Aug 11 19:22
    fanatid assigned #3422
  • Aug 11 19:22
    fanatid opened #3422
  • Aug 11 19:00
    lukesteensen synchronize #3416
  • Aug 11 17:00
    fanatid closed #3347
  • Aug 11 16:08
    ktff review_requested #3421
  • Aug 11 16:08
    ktff labeled #3421
  • Aug 11 16:08
    ktff assigned #3421
Grant Schofield
@schofield
will check back Monday
Oh and the Kafka did work fine with our own kafka connect sink, and kafkacat works...
Binary Logic
@binarylogic

Thanks @schofield, we'll see what's going on with MSK. I've opened timberio/vector#1833 to look into it.

Regarding the rate limit error, you just need to raise https://vector.dev/docs/reference/sinks/humio_logs/#rate_limit_num . Let me know if we should raise the default there.

Grant Schofield
@schofield
thanks, missed that one, will give it a go
Andrey Afoninsky
@afoninsky
just noticed:
you've added "loki" sync as a part of the sprint
and it appeared in documentation which is good: https://vector.dev/docs/reference/sinks/loki/
the problem is - there is no information that it's a part of a nightly build, and not a part of latest stable (0.7.2) so it does not work
5 replies
leidruid
@leidruid_gitlab
hello, guys! I use vector as tcp > kafka transport.
some short time after the start, sink stucks, nothing happens, no suspicious messages in broker and vector logs (in debug mode too)
for a litte time before stuck, in vector log disappears sink messages (only source messages remains)
Has anyone encountered this problem?
4 replies
elbaro
@elbaro

Hello, I am routing glog file to elasticsearch.

[transforms.add_metadata] 
  type = "lua"
  inputs = ["src"]
  source = """
event['app'] = event['file']:match('/log/(.+)%.INFO')
"""

[sinks.proj_sink]
  type = "elasticsearch"
  inputs = ["add_metadata"]
  host = "http://1.2.3.4:5"
  index = "proj-{{host}}-{{app}}"

The above snippet does not work. The log shows nothing.

However

  index = "proj-{{host}}"

works and msgs have correct app field. Why can't I use {{app}} in index?

2 replies
Samuel Cormier-Iijima
@sciyoshi

Hi all, I'm not able to get the aws_ec2_metadata transform working. It seems to be connecting correctly to the metadata endpoint, but never makes a request. Here's my sample config:

data_dir = "/var/lib/vector"

[sources.stdin]
  type = "stdin"

[transforms.add_aws_metadata]
  type = "aws_ec2_metadata"
  inputs = ["stdin"]

[sinks.console]
  type = "console"
  inputs = ["add_aws_metadata"]
  encoding = "json"

And the debug logs:

Feb 19 21:59:30.069  INFO vector: Log level "info" is enabled.
Feb 19 21:59:30.069  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Feb 19 21:59:30.072  INFO vector: Vector is starting. version="0.8.0" git_version="v0.7.0-168-g841a8f8" released="Wed, 19 Feb 2020 11:14:46 +0000" arch="x86_64"
Feb 19 21:59:30.073 DEBUG trust_dns_resolver::async_resolver::background: trust-dns resolver running
Feb 19 21:59:30.073  INFO vector::topology: Running healthchecks.
Feb 19 21:59:30.073 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 21:59:30.074  INFO vector::topology: Starting source "stdin"
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))
Feb 19 21:59:30.075  INFO vector::topology::builder: Healthcheck: Passed.
Feb 19 21:59:30.076  INFO vector::topology: Starting transform "add_aws_metadata"
Feb 19 21:59:30.076  INFO source{name=stdin type=stdin}: vector::sources::stdin: Capturing STDIN
Feb 19 21:59:30.076  INFO vector::topology: Starting sink "console"
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 4194305
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 1
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 8388610
Feb 19 21:59:30.076 DEBUG tokio_reactor::background: starting background reactor
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 2
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 12582915
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 3
Feb 19 21:59:30.077 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG hyper::proto::h1::io: flushed 102 bytes
test
{"host":"6792b4ce6d87","timestamp":"2020-02-19T21:59:37.278494604Z","message":"test"}

I'm running inside Docker using the nightly-alpine image. Any help/thoughts appreciated!

Actually, if I wait long enough I see

Feb 19 22:02:17.835 DEBUG hyper::proto::h1::conn: parse error (connection error: Connection reset by peer (os error 104)) with 0 bytes
Feb 19 22:02:17.835 DEBUG hyper::proto::h1::dispatch: read_head error: connection error: Connection reset by peer (os error 104)
Feb 19 22:02:17.835 DEBUG tokio_reactor: dropping I/O source: 0
Feb 19 22:02:17.836 ERROR aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Unable to fetch EC2 metadata; Retrying. error=connection error: Connection reset by peer (os error 104)
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))

Maybe there's something else going on - although I'm able to curl the metadata endpoint from inside the container...

Lucio Franco
@LucioFranco
What type of instance are you on? and what does your curl command look like
Samuel Cormier-Iijima
@sciyoshi
thanks for the quick response @LucioFranco! it's a standard EC2 instance, m5.xlarge. here's the command I'm running:
admin@ip-172-20-98-28:~$ sudo docker run -it --entrypoint /bin/sh -e LOG=debug --rm --name vector -v $PWD/vector.toml:/etc/vector/vector.toml -v /var/lib/docker:/var/lib/docker -v /var/run/docker.sock:/var/run/docker.sock -v
$PWD/vector:/var/lib/vector -v /var/log/pods:/var/log/pods timberio/vector:nightly-alpine
/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/3) Installing nghttp2-libs (1.39.2-r0)
(2/3) Installing libcurl (7.66.0-r0)
(3/3) Installing curl (7.66.0-r0)
Executing busybox-1.30.1-r3.trigger
OK: 10 MiB in 19 packages
/ # curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "------------",
  "architecture" : "x86_64",
  "availabilityZone" : "ca-central-1a",
  "billingProducts" : null,
  "devpayProductCodes" : null,
  "marketplaceProductCodes" : null,
  "imageId" : "ami-0xxxxxx",
  "instanceId" : "i-0xxxxxx",
  "instanceType" : "m5.xlarge",
  "kernelId" : null,
  "pendingTime" : "2020-02-11T15:42:59Z",
  "privateIp" : "172.20.98.28",
  "ramdiskId" : null,
  "region" : "ca-central-1",
  "version" : "2017-09-30"
}/
Lucio Franco
@LucioFranco
ah looks like you're running vector within a container, that may be the reason
Samuel Cormier-Iijima
@sciyoshi
the curl command is also running from within the container
Lucio Franco
@LucioFranco
@sciyoshi can you try running the docker command with --net=host?
Samuel Cormier-Iijima
@sciyoshi
oh yup, that worked!! thank you :) not sure why curl would have been able to connect?
Lucio Franco
@LucioFranco
I would assume black magic :) glad that worked! let us know if you have any other issues.
Samuel Cormier-Iijima
@sciyoshi
I have another quick question - the json_parser transform seems to always remove the source field when drop_field is true. This seems inconsistent with the behavior of e.g. grok_parser, which only removes it when the parse succeeds. Is that behavior intentional?
Binary Logic
@binarylogic
Hey @sciyoshi , the behavior should be consistent across the two. I've opened timberio/vector#1861 to fix that.
Sebastian YEPES
@syepes
Small question, Is it currently possible to ingest (receive from UDP, TCP or file) metrics using the line protocol?
2 replies
Samuel Cormier-Iijima
@sciyoshi
@LucioFranco update on the original issue - I'm not able to use --net=host, but also it seems that it's only the /latest/api/token endpoint that is timing out from within a container. It seems that the API that should be used instead is the IMDS metadata - botocore updated due to this issue and you can see the changes here: boto/botocore#1895
7 replies
Andrey Afoninsky
@afoninsky
does vector have loggly support? haven't found any issues about it: https://github.com/timberio/vector/search?q=loggly&unscoped_q=loggly
1 reply
Aleksey Shirokih
@freeseacher
Hi! how can i transform something like that "file":"/var/log/mysystem/subsystem-component_name-07.log" to component_name ?
1 reply
Ana Hobden
@Hoverbear
Glad you got it!
Aleksey Shirokih
@freeseacher
as i can see there are type https://vector.dev/docs/about/data-model/metric/#aggregated_summary but how can i get it ? i am interested in prometheus summary of cause. there are some reference to timberio/vector#710 but i can't catch the point
Ana Hobden
@Hoverbear
@freeseacher if you're taking in logs and want to output metrics please try https://vector.dev/docs/reference/transforms/log_to_metric/
Aleksey Shirokih
@freeseacher
yes i am talking about metrics and already found log2metric but it does not help. type must must be one of: "counter" "gauge" "histogram" "set" but not quantile.
Samuel Cormier-Iijima
@sciyoshi
I am having issues with Docker log rotation using the default json-file logging driver - Vector stops picking up logs after the file is rotated
25 replies
C├ędric Da Fonseca
@Kwelity
Hi, I'm not sure to understand how the regex transform works.
I'm trying to only parse error log message, so I have a regexp starting with "^ERROR.*", I'm expecting the transform to drop the log that doesn't match. But, the log is parsed and the log content is put in the "message" field.
I tried to play with drop_field and field but it didn't work
What would be the best solution for my use case ?
2 replies
Heinz N. Gies
@Licenser
it worked :D
Ana Hobden
@Hoverbear
Gitter: It works sometimes! :)
mlki
@MlkiTouch_twitter

Hello someone has try AWS S3 Sink with ceph ? For me it doesn't work, for example for the healtcheck ceph return a 404 response code for the head method while it return 200 response code when i'm using mc ls, here is the config

[sinks.ceph]
  # REQUIRED - General
  type = "aws_s3" # must be: "aws_s3"
  inputs = ["syslog"] # example
  bucket = "vector" # example
  compression = "none" # example, enum
  endpoint = "http://my-ceph.com:9000"

  # OPTIONAL - Object Names
  filename_append_uuid = true # default
  filename_extension = "log" # default
  filename_time_format = "%s" # default
  key_prefix = "date=%F/" # default
  # REQUIRED - requests
  encoding = "text" # example, enum

  # OPTIONAL - General
  healthcheck = true# default

i set also the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY . When i try to send a log it returns me
Feb 28 16:40:05.185 ERROR sink{name=ceph type=aws_s3}: vector::sinks::util::retries: encountered non-retriable error. error=<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>http://my-ceph.com:9000</BucketName><RequestId>tx00000000000000c51a948-005e594265-430c8a-myhost-1</RequestId><HostId>myhostid</HostId></Error> Feb 28 16:40:05.185 ERROR sink{name=ceph type=aws_s3}: vector::sinks::util: request failed. error=<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>http://my-ceph.com:9000</BucketName><RequestId>tx00000000000000c51a948-005e594265-430c8a-myhost-1</RequestId><HostId>myhostid</HostId></Error>
Could you help me with that please ? :-) Have a nice day

mahsoud
@mahsoud
Hey everyone, just started playing with vector agent on Windows to collect logs from a legacy application. In my case, when the application starts it writes a very long line into the log file (\u0000 on repeat)... what transform would you suggest to use to drop that one line?
Andrey Afoninsky
@afoninsky

hello
https://github.com/prometheus/statsd_exporter

Note that timers will be accepted with the ms, h, and d statsd types. The first two are timers and histograms and the d type is for DataDog's "distribution" type. The distribution type is treated identically to timers and histograms.

does vector support DD type? do we need to create issue?

2 replies
Andrey Afoninsky
@afoninsky
https://medium.com/@valyala/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350
does it make sense to create an issue with implementation request for prometheus sync?
pros: a better histogram (less cardinality, more accuracy)
cons: VictoriaMetric specific only, maybe it's useful in specific cases only
1 reply
ChethanU
@ChethanUK
Is there Offical helm chart?
2 replies
Bill
@bill-bateman

Hey - I have a small problem with reloading configurations. If the source is http / logplex / splunk_hec (all of which use Warp) and you change the configuration, but don't change the port, I get a configuration error (address already in use) and the reload fails. Workaround is to just change the port to a new value. After a successful reload you can then change the port back to the original.

It's not a huge issue, but I wanted to see if it was known.

ERROR vector::topology: Configuration error: Source "in": Address already in use (os error 48)
ERROR vector: Reload was not successful.
leidruid
@leidruid_gitlab
hello! Is there a correct way to specify multiple targets in elasticsearch sink, as in logstash?
9 replies
Andrey Afoninsky
@afoninsky
hello, please fix me if I wrong: "vector" source is a grpc-server and I can send logs/metrics directly using https://github.com/timberio/vector/blob/master/proto/event.proto ?
2 replies
Andrey Afoninsky
@afoninsky
another question: what's the best approach to implement log rotation / truncate in https://vector.dev/docs/reference/sinks/file/ and docker image? do you want to have an issue about it or it should be achieved using external tools? for now, I'm launching logrotate docker image as sidecar :)
2 replies
Andrey Afoninsky
@afoninsky
please take a look if you have a free time, can't understand is it a bug or my misunderstanding of documentation :) thx
timberio/vector#2036
2 replies
Andrey Afoninsky
@afoninsky
one more question :) "file sink" does not recreate a file if old one was deleted using "rm" - is it correct behaviour?
7 replies
gtie
@gtie
anyone else seeing behavior like timberio/vector#2080 ?
mmacedo
@mmacedoeu
hi, building vector using make or using docker as stated on https://vector.dev/docs/setup/installation/manual/from-source/ generates a debug mode binary
1 reply
is there any instruction I am missing to generate a release version ?
mmacedo
@mmacedoeu
I found that hotmic is deprecated, https://github.com/timberio/vector/blob/master/lib/tracing-metrics/Cargo.toml#L9 do you plan to replace it with crate metrics ?
6 replies
Serhii M.
@mikhno-s

Hi, guys! Small question regarding config for kafka sink - I have next piece of config

[sinks.kafka]
  type = "kafka"
  inputs = ["json"]
  bootstrap_servers = "kafka-server:9092"
  topic = "vector"
  compression = "none"
  healthcheck = true

  buffer.type = "disk"
  buffer.max_size = 104900000
  buffer.when_full = "block"

  encoding.codec = "json"

And when I try to start vector I get:

unknown variant `codec`, expected `text` or `json` for key `sinks.kafka`

What's wrong with config?

Thanks!

Serhii M.
@mikhno-s
ok, it looks like in docs is not working example - work well just with encoding = "json"
Ana Hobden
@Hoverbear
@mmacedoeu yes! @lukesteensen has a PR #1953 to do that