Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
  • 20:33
    lukesteensen synchronize #1953
  • 20:18
    lukesteensen review_requested #1953
  • 20:17
    lukesteensen synchronize #1953
  • 20:07
    lukesteensen synchronize #1953
  • 19:58
    a-rodin closed #2231
  • 19:57
    a-rodin edited #2245
  • 19:53
    a-rodin review_requested #2231
  • 19:44
    a-rodin synchronize #2231
  • 19:31
    binarylogic closed #2251
  • 19:27
    binarylogic synchronize #2251
  • 19:25
    binarylogic synchronize #2251
  • 19:23
    a-rodin closed #2252
  • 19:14
    binarylogic synchronize #2251
  • 19:13
    a-rodin review_requested #2252
  • 19:13
    a-rodin opened #2252
  • 19:09
    binarylogic synchronize #2251
  • 19:07
    binarylogic opened #2251
  • 18:48
    binarylogic closed #2249
  • 18:39
    a-rodin review_requested #2249
  • 18:39
    a-rodin opened #2249
Ana Hobden
Gitter: It works sometimes! :)

Hello someone has try AWS S3 Sink with ceph ? For me it doesn't work, for example for the healtcheck ceph return a 404 response code for the head method while it return 200 response code when i'm using mc ls, here is the config

  # REQUIRED - General
  type = "aws_s3" # must be: "aws_s3"
  inputs = ["syslog"] # example
  bucket = "vector" # example
  compression = "none" # example, enum
  endpoint = "http://my-ceph.com:9000"

  # OPTIONAL - Object Names
  filename_append_uuid = true # default
  filename_extension = "log" # default
  filename_time_format = "%s" # default
  key_prefix = "date=%F/" # default
  # REQUIRED - requests
  encoding = "text" # example, enum

  # OPTIONAL - General
  healthcheck = true# default

i set also the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY . When i try to send a log it returns me
Feb 28 16:40:05.185 ERROR sink{name=ceph type=aws_s3}: vector::sinks::util::retries: encountered non-retriable error. error=<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>http://my-ceph.com:9000</BucketName><RequestId>tx00000000000000c51a948-005e594265-430c8a-myhost-1</RequestId><HostId>myhostid</HostId></Error> Feb 28 16:40:05.185 ERROR sink{name=ceph type=aws_s3}: vector::sinks::util: request failed. error=<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>http://my-ceph.com:9000</BucketName><RequestId>tx00000000000000c51a948-005e594265-430c8a-myhost-1</RequestId><HostId>myhostid</HostId></Error>
Could you help me with that please ? :-) Have a nice day

Hey everyone, just started playing with vector agent on Windows to collect logs from a legacy application. In my case, when the application starts it writes a very long line into the log file (\u0000 on repeat)... what transform would you suggest to use to drop that one line?
Andrey Afoninsky


Note that timers will be accepted with the ms, h, and d statsd types. The first two are timers and histograms and the d type is for DataDog's "distribution" type. The distribution type is treated identically to timers and histograms.

does vector support DD type? do we need to create issue?

Andrey Afoninsky
does it make sense to create an issue with implementation request for prometheus sync?
pros: a better histogram (less cardinality, more accuracy)
cons: VictoriaMetric specific only, maybe it's useful in specific cases only
Is there Offical helm chart?

Hey - I have a small problem with reloading configurations. If the source is http / logplex / splunk_hec (all of which use Warp) and you change the configuration, but don't change the port, I get a configuration error (address already in use) and the reload fails. Workaround is to just change the port to a new value. After a successful reload you can then change the port back to the original.

It's not a huge issue, but I wanted to see if it was known.

ERROR vector::topology: Configuration error: Source "in": Address already in use (os error 48)
ERROR vector: Reload was not successful.
hello! Is there a correct way to specify multiple targets in elasticsearch sink, as in logstash?
Andrey Afoninsky
hello, please fix me if I wrong: "vector" source is a grpc-server and I can send logs/metrics directly using https://github.com/timberio/vector/blob/master/proto/event.proto ?
Andrey Afoninsky
another question: what's the best approach to implement log rotation / truncate in https://vector.dev/docs/reference/sinks/file/ and docker image? do you want to have an issue about it or it should be achieved using external tools? for now, I'm launching logrotate docker image as sidecar :)
Andrey Afoninsky
please take a look if you have a free time, can't understand is it a bug or my misunderstanding of documentation :) thx
Andrey Afoninsky
one more question :) "file sink" does not recreate a file if old one was deleted using "rm" - is it correct behaviour?
anyone else seeing behavior like timberio/vector#2080 ?
hi, building vector using make or using docker as stated on https://vector.dev/docs/setup/installation/manual/from-source/ generates a debug mode binary
is there any instruction I am missing to generate a release version ?
I found that hotmic is deprecated, https://github.com/timberio/vector/blob/master/lib/tracing-metrics/Cargo.toml#L9 do you plan to replace it with crate metrics ?
Serhii M.

Hi, guys! Small question regarding config for kafka sink - I have next piece of config

  type = "kafka"
  inputs = ["json"]
  bootstrap_servers = "kafka-server:9092"
  topic = "vector"
  compression = "none"
  healthcheck = true

  buffer.type = "disk"
  buffer.max_size = 104900000
  buffer.when_full = "block"

  encoding.codec = "json"

And when I try to start vector I get:

unknown variant `codec`, expected `text` or `json` for key `sinks.kafka`

What's wrong with config?


Serhii M.
ok, it looks like in docs is not working example - work well just with encoding = "json"
Ana Hobden
@mmacedoeu yes! @lukesteensen has a PR #1953 to do that
@mikhno-s yes unfortunately the docs are showing a new feature we're about to release
How do people monitor vector in production? Figuring out that the service is up fine, but how can you tell if it is indeed capable of shipping data to its sink(s)?
Is timber.io still being supported? The site is currently broken for me after logging in.

I'm having issues with
unknown variant `codec`, expected `text` or `json` for key `sinks.some_sink` for several different types of sinks... It only works when specifying encoding = "text" or encoding = "json" - Problem is, I need some of the options under encoding.

Tried looking at the source, but I'm not familiar with Rust enough to locate the error myself.

Anyone know if this is a known bug?

Andrey Afoninsky
is there a way to trigger health check periodically? will "vector --dry-run --require-healthy --quiet" do the job?
Is it possible to route log stream to a specific Splunk index using splunk-hec sink?
In fluent* it's done by adding "index" field and enabling "send_raw" option. However I couldn't find any example for vector.
Madhurranjan Mohaan
Hi, Is there anyone using vector to stream logs from envoy logs and upload it to S3 or GCS ?
Chris Holcombe
Hi everyone. I was thinking of giving vector a try but I'm in need of some clarification. It looks like there's a required schema for every log event. Is that correct?
Madhurranjan Mohaan
Hi, what is the limit recommended in terms of bytes per record? On the website it says, its not a replacement for an analytics record. How do you define an analytics record ? Based on bytes / no of fields / something else ?
hey folks, I 'm new to Vector, just a quick question, can Vector output to AWS SQS?
Am I right if the Vector is log shipper like Filebeat?
Serhii M.

Hi, everybody
Does anybody know what TRACE sink{name=s3_logs type=aws_s3}: tower_limit::rate::service: rate limit exceeded, disabling service means?

I don't see any other errors in logs. However I see, that vector reads the log files, but does not send any to s3 :(

Binary Logic
Hi @mikhno-s , that log indicates internal rate limiting. You can raise the defaults here: https://vector.dev/docs/reference/sinks/aws_s3/#request. For example, if you bump rate_limit_num that will allow more throughput.
Иван Афанасьев
I am using sources = file. My files are in ANSI encoding and contain Russian characters. I try to apply regexp, but it does not work out properly.
If the file is encoded in utf8 and contains Russian characters, regexp works as it should.
Can I specify CHARSET for sources = file?
Can I convert a string in a Transforms block?
Can we use TAB as separator? The example at https://vector.dev/docs/reference/transforms/split/#separator is not clear. Will something like separator = "\t" work?
Or if I use tokenizer to split the fields https://vector.dev/docs/reference/transforms/tokenizer/ , can tokenizer use TAB?
Another question: is there a way to define the flush interval? eg: I have "socket" as sources and "file" as sinks, can I "create the file" (flush out the source to the file) every 30 seconds?
Иван Афанасьев
Is it possible, when using sources = file, to skip the first N lines from each new file since they have a description of fields?
@LucioFranco Thanks for your comments in timberio/vector#2174 Regarding your suggestion of using disk buffer with gcs sink, I have a few questions: 1) any flakiness on the WAN and sink side should not cause any issues on the source socket receiving? 2) for batch, I guess max_size/timeout_secs is whichever comes first, and when max_size reaches, it will reset the time for timeout? 3) Can we control how filename look like when sending to gcs? eg: use one field/tag from the input line, so that different input lines can be stored in its respective files at a certain time interval?
Zahav Capper

Hi there,
Has anyone here successfully configured vector to ship to Amazon Elasticsearch?
(I believe) I've configured the EC2 instance profiles and Elasticsearch permissions correctly but I'm getting a 403 in the logs:
Mar 31 10:42:07.843 WARN sink{name=elasticsearch_vpcflowlogs type=elasticsearch}: vector::sinks::util::retries: request is not retryable; dropping the request. reason=response status: 403 Forbidden

Not sure where to start looking to debug this

Rick Richardson
what is the recommended configuration for general k8s logging with vector? I am assuming that docker log-driver=journald would be the simplest..
Brad Fritz

Thank you for releasing armv7 binaries. The current build setup makes it pretty easy to also build for armv6 (arm-unknown-linux-musleabihf) which is needed to run on Raspberry Pi Zero.

Any interest in supporting armv6 officially? If so, I can submit a PR.

Ana Hobden
@bfritz Hey! :) Hm, you're right! Have you been already using it? How does it run? Some of our upcoming features might be hard to support on armv6, would you be ok with a reduced feature version if needed?
Ana Hobden
@bfritz Definitely open and issue and we can advocate for this feature. :)
I opened timberio/vector#2243 and am wondering if there is already discussion on this?
Martin Grünbaum
Hi there - I've got a Kafka source that works, and a console sink that works. I've added a GCS sink too, which authenticates -- however, I don't see any files in the bucket pop up; nor any log messages by Vector, even if I move it to DEBUG log level.
The console sink outputs the message on the topic, but theres zero log output related to the GCS sink when I send a message to the topic
Martin Grünbaum
Nevermind - I'm an idjit, batch settings >.>
Martin Grünbaum

Hmm, okay - stuck again.

I'm getting a '400 Bad Request' from GCP on my GCS sink, but even on TRACE level it's not showing the body of the response so I can't get at which actual problem it's encountering. All the output I get on trace is:

Apr 07 13:50:06.853 TRACE sink{name=gcp type=gcp_cloud_storage}: vector::sinks::util: request succeeded. response=Response { status: 400, version: HTTP/1.1, headers: {"x-guploader-uploadid": "xxx", "content-type": "application/xml; charset=UTF-8", "content-length": "170", "vary": "Origin", "date": "Tue, 07 Apr 2020 13:50:06 GMT", "server": "UploadServer", "alt-svc": "quic=\":443\"; ma=2592000; v=\"46,43\",h3-Q050=\":443\"; ma=2592000,h3-Q049=\":443\"; ma=2592000,h3-Q048=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,h3-T050=\":443\"; ma=2592000"}, body: Body(Streaming) }

The body property there doesn't get revealed further down in the log anywhere, and then the connection closes.

Martin Grünbaum

It seems like the http connection is being closed by the caller before the body can be received fully? :s

Apr 07 14:05:54.397 TRACE hyper::proto::h1::dispatch: body receiver dropped before eof, closing Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: State::close_read() Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: State::close() Apr 07 14:05:54.397 TRACE tokio_threadpool::worker: -> wakeup; idx=3 Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }

Martin Grünbaum
Unfortunately I don't see any way to debug this :/ Does anyone have tips?