Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
  • 01:26
    bruceg review_requested #9308
  • 01:23
    bruceg synchronize #9308
  • Sep 27 23:45
    bruceg review_requested #9358
  • Sep 27 23:45
    bruceg labeled #9358
  • Sep 27 23:45
    bruceg review_requested #9358
  • Sep 27 23:45
    bruceg labeled #9358
  • Sep 27 23:45
    bruceg assigned #9358
  • Sep 27 23:45
    bruceg opened #9358
  • Sep 27 22:29
    jszwedko synchronize #8717
  • Sep 27 22:19
    jszwedko synchronize #8717
  • Sep 27 22:18
    001wwang closed #9318
  • Sep 27 22:09
    001wwang synchronize #9318
  • Sep 27 22:01
    001wwang synchronize #9318
  • Sep 27 21:36
    001wwang synchronize #9318
  • Sep 27 21:29
    jszwedko closed #9015
  • Sep 27 21:23
    jszwedko edited #9330
  • Sep 27 21:21
    jszwedko opened #9357
  • Sep 27 21:21
    jszwedko review_requested #9357
  • Sep 27 21:18
    jszwedko auto_merge_enabled #9015
  • Sep 27 20:34
    jszwedko review_requested #8717
the logs are showing up in s3 with .gz extension, however they are still plain text files
have anyone exeperienced something like this and maybe found a solution ?
Slawomir Skowron
If you download through the browser files may be decompresed on the fly. You can compare dowload size vs size reported on s3.
Ryan Miguel
If you need TLS please vote for this issue: timberio/vector#3664
Jesse Szwedko


Hey all!

A quick announcement: we are moving from gitter to discord for our community chat. You can join us here: https://discord.gg/jm97nzy (see channels in the vector category).


As the team supporting vector and building its community, we've found a number of issues using gitter for this purpose:

  • Poor notifications
  • Poor editing experience
  • Poor mobile support

We hope having people come to discord instead will result in more messages being seen and responded to.

We also hope to move more of our general development discussions to discord as well to make it easier for people to follow along and contribute.

For more detailed support issues, Github Issues is still the best place to ensure that the they are seen, triaged, and responded to.

The link on the website and other pointers will be updated shortly.

Hope to see you there! We also welcome any feedback on how we can better support the vector community.

abbas ali chezgi
please correct this link on github issue reporting page: https://github.com/timberio/vector/issues/new/choose
1 reply
I'm evaluating Vector to replace fluent, one thing I have noticed is the absolute write limit to s3 is significantly slower than fluentd. Is there a way to improve throughput ?
3 replies
Michael Pietzsch
Hi Guys, i got my Vector setup running today. I got a syslog source pushing in to a loki sink. But i am struggling to setup static labels Im only getting a "agent="vector" label in grafana
1 reply
Andrey Afoninsky

a generic question about periodic health check:

  • we have "--require-healthy" to check problem on startup
  • we have unit tests to assist in the development of complex topology

recently, our kafka instance (sink) was down and errors started to appear in the console -> so the service stopped to work but didn't fall
it fell only after restart as "--require-healthy" flag is specified and sink is not healthy

there was a command we could trigger periodically which returned >0 exit code if health didn't pass -> but it was removed in the latest versions
a generic question: is it possible to setup health check (ex.: in kubernetes) somehow, any workarounds? thanks

5 replies
Grant Isdale

Hey all,

Does vector support the Web Identity Provider in STS? This feature was merged into Rusoto in Dec '19 (rusoto/rusoto#1577), but I'm struggling to implement.

As far as I'm aware, everything is set up correctly and Web Identity Provider works with our other k8s services (and my set-up confirmed by this guide here: https://dev.to/pnehrer/a-story-of-rusty-containers-queues-and-the-role-of-assumed-identity-kl2) but when I'm trying to put to a CloudWatch log group it won't assume the correct SA.

2 replies
Liran Albeldas
If I have multiple sinks and 1 of them getting time out. all the other stops operating until all sinks are work?
1 reply
Vyacheslav Rakhinskiy
Hi, how I can use custom grok patterns? for example https://github.com/padusumilli/postfix-grok/blob/master/postfix-grok-patterns
1 reply
Mark Klass
Hello, is there a way to use Vector's transformation to "clean" the fields? For example, I used the tokenize transformation to get some of the values from some log, but now, I have values like {"protocol":":udp", "source_port":":57714->", etc}
Is there a way to clean them? Like removing the : in protocol and the : and -> in source_port?
Hi I am using http source and when I start and stop my application multiple times(which restart vector as well each time), on client side I start getting error in connecting to vector: Error message:Connection refused (Connection refused)
i'm tringto use vecto on my docker compose
but it can't catch the traffic on 5000 port
on my local machine
Jesse Szwedko
Hi @Bindu-Mawat and @mikele_g_twitter ; just a note that community discussion and support has moved to discord: https://discord.gg/jm97nzy
Vyacheslav Rakhinskiy
HI, how I can disable rate_limits on errors (I set drop_invalid to true for json_parser)?
For example
Nov 25 12:11:45.976 WARN transform{name=nginx_parse_json type=json_parser}: vector::internal_events::json: 19 "Event failed to parse as JSON" events were rate limited. rate_limit_secs=5
Vyacheslav Rakhinskiy
And how I can debug this json? I start vector with LOG="trace" and can see only WARN for this event
Jesse Szwedko
Hi @rakhinskiy ! Just a note that community discussion and support has moved to discord: https://discord.gg/jm97nzy
Vyacheslav Rakhinskiy
@jszwedko ok thanks
Liem Le Hoang Duc

Hi there, I'm stuck at write log to file with correct timezone.
inputs = ["in"]
type = "file"
path = "/tmp/%Y-%m-%d/%H.log"
encoding.codec = "text"

The time is UTC based where I need it in Local Timezone (+7 in my case). Is there anyway to achieve this with Vector? I've searched around but no luck.

Jesse Szwedko
Hi @liemle3893 ! Just a note that community discussion and support has moved to discord: https://discord.gg/jm97nzy
1 reply
Hey, guys. Does the vector is fips compatible?
Hi @valerypetrov ! Just a note that community discussion and support has moved to discord: https://discord.gg/jm97nzy
I have error unknown field transform
type = "remap"
inputs = ["import_logs_tr"]
source = '''
.del(.file, .host)
.log = "import"
this transform is as in docs
data before transform is:
{"file":"/var/log/app/app.log","host":"244f68ac3445","message":"num **213","severity":"import","source_type":"file","timestamp":"2021-05-19T13:06:18Z"}
Sanskar Gupta

I have a rust tcp server running at 4000, when I try to start the vector to get the tcp metrics am getting

ERROR source{component_kind="source" component_name=my_source_id component_type=socket}: vector::sources::util::tcp: Failed to bind to listener socket. error=TCP bind failed: Address already in use (os error 98)
Jun 23 22:47:43.438 ERROR source{component_kind="source" component_name=my_source_id component_type=socket}: vector::topology: An error occurred that vector couldn't handle.
Here is my vector.toml

type = "socket"
address = ""
mode = "tcp"

inputs = ["my_source_id"]
type = "console"
encoding = "text"


anyone have an example of a config for getting pod logs from a specific kubernetes namespace only? i'm trying

    type: kubernetes_logs
    extra_field_selector: metadata.namespace==vector-testing

    type: filter
      - kube_log_source
    condition: .kubernetes.namespace == "vector-testing"

to no avail

Pasha Radchenko

Guys, could you please help me with the TOML configuration for vector?
I have the following configmap, trying to implement multiline for java stack traces:

 18     [transforms.my_custom_k8s_multiline]
 19     type = "reduce"
 20     inputs = [ "kuber" ]
 21     group_by = [ "file", "kubernetes", "source_type", "stream", "timestamp" ]
 22     starts_when.type = "vrl"
 23     starts_when.source = "match(.message, r'^\\[')"
 24     expire_after_ms = 3_000
 25     flush_period_ms = 300
 27       [transforms.my_custom_k8s_multiline.merge_strategies]
 28       message = "concat_newline"

but it fails with something like

 ~ k logs vector-mx4h2
Sep 23 15:10:13.857  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info" enable_datadog_tracing=false
Sep 23 15:10:13.858  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
Sep 23 15:10:13.862  INFO vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="ip-10-221-172-92.ec2.internal"
Sep 23 15:10:13.885  INFO vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["/var/log/pods/cluster-*/**", "/var/log/pods/kube-*/**", "/var/log/pods/*_nginx-*/**", "/var/log/pods/logging_*/**"]
Sep 23 15:10:13.912 ERROR vector::topology: Configuration error. error=Transform "my_custom_k8s_multiline":
error[E100]: unhandled error
  ┌─ :1:11 │ match(.message, r'^\[')
  │ ^^^^^^^^^^^^^^^^^^^^^^^
  │ │
  │ expression can result in runtime error
  │ handle the error case to ensure runtime success

I've tried a lot of variants with match function arguments, remove and revert quotes but still no luck. Really appreciate if somebody can look into :)

Ajay Gupta
Hi Can we write custom plugin in vector?