by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 11:05
    fanatid review_requested #2993
  • 11:05
    fanatid review_requested #2993
  • 11:04
    fanatid review_requested #2993
  • 11:04
    fanatid labeled #2993
  • 11:04
    fanatid labeled #2993
  • 11:04
    fanatid labeled #2993
  • 11:04
    fanatid labeled #2993
  • 11:04
    fanatid labeled #2993
  • 11:04
    fanatid opened #2993
  • 00:22
    MOZGIII closed #2975
  • Jul 08 23:57
    Hoverbear closed #2982
  • Jul 08 23:50
    Hoverbear labeled #2982
  • Jul 08 23:29
    Hoverbear labeled #2987
  • Jul 08 23:12
    Hoverbear closed #2990
  • Jul 08 23:11
    Hoverbear closed #2988
  • Jul 08 23:10
    Hoverbear labeled #2988
  • Jul 08 23:09
    dependabot-preview[bot] synchronize #2987
  • Jul 08 23:09
    dependabot-preview[bot] edited #2987
  • Jul 08 23:09
    Hoverbear labeled #2967
  • Jul 08 23:08
    dependabot-preview[bot] edited #2987
Ana Hobden
@Hoverbear
@mikhno-s yes unfortunately the docs are showing a new feature we're about to release
gtie
@gtie
How do people monitor vector in production? Figuring out that the service is up fine, but how can you tell if it is indeed capable of shipping data to its sink(s)?
3 replies
carumusan
@carumusan
Is timber.io still being supported? The site is currently broken for me after logging in.
1 reply
Mads
@MadsRC_gitlab

I'm having issues with
unknown variant `codec`, expected `text` or `json` for key `sinks.some_sink` for several different types of sinks... It only works when specifying encoding = "text" or encoding = "json" - Problem is, I need some of the options under encoding.

Tried looking at the source, but I'm not familiar with Rust enough to locate the error myself.

Anyone know if this is a known bug?

1 reply
Andrey Afoninsky
@afoninsky
is there a way to trigger health check periodically? will "vector --dry-run --require-healthy --quiet" do the job?
1 reply
Alex
@Alexx-G
Hi,
Is it possible to route log stream to a specific Splunk index using splunk-hec sink?
In fluent* it's done by adding "index" field and enabling "send_raw" option. However I couldn't find any example for vector.
Thanks.
8 replies
Madhurranjan Mohaan
@madhurranjan_twitter
Hi, Is there anyone using vector to stream logs from envoy logs and upload it to S3 or GCS ?
2 replies
Chris Holcombe
@cholcombe973
Hi everyone. I was thinking of giving vector a try but I'm in need of some clarification. It looks like there's a required schema for every log event. Is that correct?
4 replies
Madhurranjan Mohaan
@madhurranjan_twitter
Hi, what is the limit recommended in terms of bytes per record? On the website it says, its not a replacement for an analytics record. How do you define an analytics record ? Based on bytes / no of fields / something else ?
1 reply
Pasha Radchenko
@ep4sh
hey folks, I 'm new to Vector, just a quick question, can Vector output to AWS SQS?
1 reply
Am I right if the Vector is log shipper like Filebeat?
2 replies
Serhii M.
@mikhno-s

Hi, everybody
Does anybody know what TRACE sink{name=s3_logs type=aws_s3}: tower_limit::rate::service: rate limit exceeded, disabling service means?

I don't see any other errors in logs. However I see, that vector reads the log files, but does not send any to s3 :(

15 replies
Binary Logic
@binarylogic
Hi @mikhno-s , that log indicates internal rate limiting. You can raise the defaults here: https://vector.dev/docs/reference/sinks/aws_s3/#request. For example, if you bump rate_limit_num that will allow more throughput.
Иван Афанасьев
@IRelajado_twitter
Hello!
I am using sources = file. My files are in ANSI encoding and contain Russian characters. I try to apply regexp, but it does not work out properly.
If the file is encoded in utf8 and contains Russian characters, regexp works as it should.
Can I specify CHARSET for sources = file?
Can I convert a string in a Transforms block?
2 replies
gfrankliu
@gfrankliu
Can we use TAB as separator? The example at https://vector.dev/docs/reference/transforms/split/#separator is not clear. Will something like separator = "\t" work?
gfrankliu
@gfrankliu
Or if I use tokenizer to split the fields https://vector.dev/docs/reference/transforms/tokenizer/ , can tokenizer use TAB?
4 replies
gfrankliu
@gfrankliu
Another question: is there a way to define the flush interval? eg: I have "socket" as sources and "file" as sinks, can I "create the file" (flush out the source to the file) every 30 seconds?
2 replies
Иван Афанасьев
@IRelajado_twitter
Hey.
Is it possible, when using sources = file, to skip the first N lines from each new file since they have a description of fields?
2 replies
gfrankliu
@gfrankliu
@LucioFranco Thanks for your comments in timberio/vector#2174 Regarding your suggestion of using disk buffer with gcs sink, I have a few questions: 1) any flakiness on the WAN and sink side should not cause any issues on the source socket receiving? 2) for batch, I guess max_size/timeout_secs is whichever comes first, and when max_size reaches, it will reset the time for timeout? 3) Can we control how filename look like when sending to gcs? eg: use one field/tag from the input line, so that different input lines can be stored in its respective files at a certain time interval?
1 reply
Zahav Capper
@zcapper

Hi there,
Has anyone here successfully configured vector to ship to Amazon Elasticsearch?
(I believe) I've configured the EC2 instance profiles and Elasticsearch permissions correctly but I'm getting a 403 in the logs:
Mar 31 10:42:07.843 WARN sink{name=elasticsearch_vpcflowlogs type=elasticsearch}: vector::sinks::util::retries: request is not retryable; dropping the request. reason=response status: 403 Forbidden

Not sure where to start looking to debug this

6 replies
Rick Richardson
@rrichardson
what is the recommended configuration for general k8s logging with vector? I am assuming that docker log-driver=journald would be the simplest..
24 replies
Brad Fritz
@bfritz

Thank you for releasing armv7 binaries. The current build setup makes it pretty easy to also build for armv6 (arm-unknown-linux-musleabihf) which is needed to run on Raspberry Pi Zero.

Any interest in supporting armv6 officially? If so, I can submit a PR.

Ana Hobden
@Hoverbear
@bfritz Hey! :) Hm, you're right! Have you been already using it? How does it run? Some of our upcoming features might be hard to support on armv6, would you be ok with a reduced feature version if needed?
3 replies
Ana Hobden
@Hoverbear
@bfritz Definitely open and issue and we can advocate for this feature. :)
gfrankliu
@gfrankliu
I opened timberio/vector#2243 and am wondering if there is already discussion on this?
Martin Grünbaum
@alathon
Hi there - I've got a Kafka source that works, and a console sink that works. I've added a GCS sink too, which authenticates -- however, I don't see any files in the bucket pop up; nor any log messages by Vector, even if I move it to DEBUG log level.
The console sink outputs the message on the topic, but theres zero log output related to the GCS sink when I send a message to the topic
Martin Grünbaum
@alathon
Nevermind - I'm an idjit, batch settings >.>
Martin Grünbaum
@alathon

Hmm, okay - stuck again.

I'm getting a '400 Bad Request' from GCP on my GCS sink, but even on TRACE level it's not showing the body of the response so I can't get at which actual problem it's encountering. All the output I get on trace is:

Apr 07 13:50:06.853 TRACE sink{name=gcp type=gcp_cloud_storage}: vector::sinks::util: request succeeded. response=Response { status: 400, version: HTTP/1.1, headers: {"x-guploader-uploadid": "xxx", "content-type": "application/xml; charset=UTF-8", "content-length": "170", "vary": "Origin", "date": "Tue, 07 Apr 2020 13:50:06 GMT", "server": "UploadServer", "alt-svc": "quic=\":443\"; ma=2592000; v=\"46,43\",h3-Q050=\":443\"; ma=2592000,h3-Q049=\":443\"; ma=2592000,h3-Q048=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,h3-T050=\":443\"; ma=2592000"}, body: Body(Streaming) }

The body property there doesn't get revealed further down in the log anywhere, and then the connection closes.

Martin Grünbaum
@alathon

It seems like the http connection is being closed by the caller before the body can be received fully? :s

Apr 07 14:05:54.397 TRACE hyper::proto::h1::dispatch: body receiver dropped before eof, closing Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: State::close_read() Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: State::close() Apr 07 14:05:54.397 TRACE tokio_threadpool::worker: -> wakeup; idx=3 Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }

Martin Grünbaum
@alathon
Unfortunately I don't see any way to debug this :/ Does anyone have tips?
21 replies
Martin Grünbaum
@alathon
I've got a Kafka source consuming from a large number of topics, but I don't have the option to really alter the Kafka messages themselves. They don't set a useful key, nor do the messages contain data on which topic they were sent to. This is problematic, because I'd like to sink them into something named after the topic they were consumed from -- For example in a GCS sink with a filename based partially on the topic. But I don't see that being exposed as a variable? (the topic a message was consumed from)
6 replies
occasionallydavid
@occasionallydavid
Hey there. Are nightly binaries maintained anywhere? I'd like to use the http source but not so much that I'd build it myself :)
Binary Logic
@binarylogic
[Ben Johnson, Timber] Hey David, you should be able to download nightlies here: https://vector.dev/releases/latest/download/. Let me know if you have trouble.
Andrey Afoninsky
@afoninsky

hello

maybe I'm a bit biased but here some quick thoughts about prometheus source timberio/vector#991 (it's not worth creating separate issue, just a small feedback based on usage :)

  • 90% of prometheus endpoints require autodiscovery (various orchestration systems like k8s, etc - most used cases) and it's not supported in vector
  • in most of cases it's better to use native collectors (or compatible ones like https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent) and receive benefits - delivery guarantees, deduplication etc. (especially considering that vector native proto does not have delivery guarantees so far)
  • the real "prometheus source" thing I miss: ability to fetch metric from prometheus tsdb (or compatible like victoriametrics, thanos, ...) using http api or "remote_read" protocol ... in this case it's possible to make amazing "pull-push conversion" stuff like sending metrics from prometheus to statuspage.io (prometheus->transforms->http) or more complex pipelines (had to create additional services to implement it)

I'm not saying that this source is not good, just it has a place where to grow :)

3 replies
Martin Grünbaum
@alathon
I can see a set of nightly tags on Dockerhub here: https://hub.docker.com/r/timberio/vector/tags -- but they're 8 days old. Seems nightlies stopped building 8 days ago. Any chance of getting a newer nightly?
1 reply
occasionallydavid
@occasionallydavid

I have an Nginx config producing about 1MB/sec of JSON (per node), the JSON is in array format, and parse_json does not like arrays. So far I am using add_fields + templating to wrap the array into an object literal before passing to parse_json, which is obviously nonsense, but it works.

The next problem is having parsed the array, how to expand it into an object with named properties. Is Lua the best option here? I thought Vector had a built-in "zip" transform, but seems it only supports it in combination with string splitting (using split's field_names argument)

finally, it probably falls into the "bloat" department, but has there been any thought to something like a streaming HTTP POST src/sink or e.g. WebSocket src/sink? I could make HTTP sink's batch_size arbitrarily small, but I'm worried about side effects of doing that (not least, massive overhead introduced by HTTP headers and/or new TCP connections)
it seems odd to have this fast and slick transform binary that is forced to do many heavy roundtrips in the last hurdle :) I know there is plain socket available, but that is not something I want to expose on this network
1 reply
Kris Reeves
@myndzi
Hi there. Really cool project, excited to explore it :) I was reading the docs and was a little surprised to see that the http source is "best effort" while the splunk hec source is "at least once". It doesn't seem as though Splunk's HEC acknowledgement protocol is implemented, so I'm wondering what the difference is?
6 replies
(and on a similar topic, are there any plans to implement HEC acks?)
3 replies
Kris Reeves
@myndzi
Got one more question, related to content transformation. For things like referer links in web logs, we use Unilog to perform redaction of data before it hits disk. I can see that LUA stuff would maybe suffice here, or maybe some of the other transforms. However, with a big list of query string keys, we found significant performance problems which we were able to improve by using the aho-corasick algorithm to match the (fixed) list of query string values efficiently against the log data. It seems like there exists an implementation of this in LUA, but I'm not entirely sure what to expect for performance or managing non-trivial code. Any advice on what'd be most suitable here to replace that functionality?
2 replies
Luca Palmieri
@LukeMathWalker
Hi everyone!
I have been experimenting with Vector to ship structured logs from a Rust backend service - so far so good, I managed to get a setup that satisfies all my requirements.
A dilemma comes up when I need to deploy that bad boy as a Kubernetes pod. I have checked the docs, but I haven't found an answer (or a clear cut one): what is the recommended strategy to scrape logs from a container running in a k8s pod using vector?
Should vector run in a separate container on the same pod, sharing a volume, using the file source? What are the other available options?
5 replies
Ana Hobden
@Hoverbear
@LukeMathWalker Hey! :) I think using the docker source will work ok? Since this is a service you made you can just drop Vector right into the docker container
24 replies
Andrey Afoninsky
@afoninsky

hello

I want to implement docker proxy to connect Vector to MQTT (as a replacement of Kafka in event-based architecture)
which transport it's better to choose for communication between vector container and proxy container?
"vector source/sink" looks like a native solution, but "http source/sink" has at-least-once guarantee

Ana Hobden
@Hoverbear
So Vector <-> Vector may change in the future to include other APIs like timberio/vector#2003
Where HTTP would be passing just the logs
So if you want a stable API into your MQTT probably HTTP?
1 reply
Andrey Afoninsky
@afoninsky
true
on the other hand, I hope to replace this hack with native vector mqtt support later :)
plus, need to implement additional http logic: bulks, retries, etc...
but got it, thank you for the information
Ana Hobden
@Hoverbear
Ah yeah, gotcha. :) That'd be nice to have for sure