vector/community

Discussions moved to Discord: https://discord.gg/jm97nzy

Rick Richardson

@rrichardson

what is the recommended configuration for general k8s logging with vector? I am assuming that docker log-driver=journald would be the simplest..

24 replies

Brad Fritz

@bfritz

Thank you for releasing armv7 binaries. The current build setup makes it pretty easy to also build for armv6 (arm-unknown-linux-musleabihf) which is needed to run on Raspberry Pi Zero.

Any interest in supporting armv6 officially? If so, I can submit a PR.

Ana Hobden

@Hoverbear

@bfritz Hey! :) Hm, you're right! Have you been already using it? How does it run? Some of our upcoming features might be hard to support on armv6, would you be ok with a reduced feature version if needed?

3 replies

Ana Hobden

@Hoverbear

@bfritz Definitely open and issue and we can advocate for this feature. :)

gfrankliu

@gfrankliu

I opened timberio/vector#2243 and am wondering if there is already discussion on this?

Martin Grünbaum

@alathon

Hi there - I've got a Kafka source that works, and a console sink that works. I've added a GCS sink too, which authenticates -- however, I don't see any files in the bucket pop up; nor any log messages by Vector, even if I move it to DEBUG log level.

The console sink outputs the message on the topic, but theres zero log output related to the GCS sink when I send a message to the topic

Martin Grünbaum

@alathon

Nevermind - I'm an idjit, batch settings >.>

Martin Grünbaum

@alathon

Hmm, okay - stuck again.

I'm getting a '400 Bad Request' from GCP on my GCS sink, but even on TRACE level it's not showing the body of the response so I can't get at which actual problem it's encountering. All the output I get on trace is:

Apr 07 13:50:06.853 TRACE sink{name=gcp type=gcp_cloud_storage}: vector::sinks::util: request succeeded. response=Response { status: 400, version: HTTP/1.1, headers: {"x-guploader-uploadid": "xxx", "content-type": "application/xml; charset=UTF-8", "content-length": "170", "vary": "Origin", "date": "Tue, 07 Apr 2020 13:50:06 GMT", "server": "UploadServer", "alt-svc": "quic=\":443\"; ma=2592000; v=\"46,43\",h3-Q050=\":443\"; ma=2592000,h3-Q049=\":443\"; ma=2592000,h3-Q048=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,h3-T050=\":443\"; ma=2592000"}, body: Body(Streaming) }

The body property there doesn't get revealed further down in the log anywhere, and then the connection closes.

Martin Grünbaum

@alathon

It seems like the http connection is being closed by the caller before the body can be received fully? :s

Apr 07 14:05:54.397 TRACE hyper::proto::h1::dispatch: body receiver dropped before eof, closing Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: State::close_read() Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: State::close() Apr 07 14:05:54.397 TRACE tokio_threadpool::worker: -> wakeup; idx=3 Apr 07 14:05:54.397 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }

Martin Grünbaum

@alathon

Unfortunately I don't see any way to debug this :/ Does anyone have tips?

21 replies

Martin Grünbaum

@alathon

I've got a Kafka source consuming from a large number of topics, but I don't have the option to really alter the Kafka messages themselves. They don't set a useful key, nor do the messages contain data on which topic they were sent to. This is problematic, because I'd like to sink them into something named after the topic they were consumed from -- For example in a GCS sink with a filename based partially on the topic. But I don't see that being exposed as a variable? (the topic a message was consumed from)

6 replies

occasionallydavid

@occasionallydavid

Hey there. Are nightly binaries maintained anywhere? I'd like to use the http source but not so much that I'd build it myself :)

Binary Logic

@binarylogic

[Ben Johnson, Timber] Hey David, you should be able to download nightlies here: https://vector.dev/releases/latest/download/. Let me know if you have trouble.

Andrey Afoninsky

@afoninsky

hello

maybe I'm a bit biased but here some quick thoughts about prometheus source timberio/vector#991 (it's not worth creating separate issue, just a small feedback based on usage :)

90% of prometheus endpoints require autodiscovery (various orchestration systems like k8s, etc - most used cases) and it's not supported in vector
in most of cases it's better to use native collectors (or compatible ones like https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent) and receive benefits - delivery guarantees, deduplication etc. (especially considering that vector native proto does not have delivery guarantees so far)
the real "prometheus source" thing I miss: ability to fetch metric from prometheus tsdb (or compatible like victoriametrics, thanos, ...) using http api or "remote_read" protocol ... in this case it's possible to make amazing "pull-push conversion" stuff like sending metrics from prometheus to statuspage.io (prometheus->transforms->http) or more complex pipelines (had to create additional services to implement it)

I'm not saying that this source is not good, just it has a place where to grow :)

3 replies

Martin Grünbaum

@alathon

I can see a set of nightly tags on Dockerhub here: https://hub.docker.com/r/timberio/vector/tags -- but they're 8 days old. Seems nightlies stopped building 8 days ago. Any chance of getting a newer nightly?

1 reply

occasionallydavid

@occasionallydavid

I have an Nginx config producing about 1MB/sec of JSON (per node), the JSON is in array format, and parse_json does not like arrays. So far I am using add_fields + templating to wrap the array into an object literal before passing to parse_json, which is obviously nonsense, but it works.

The next problem is having parsed the array, how to expand it into an object with named properties. Is Lua the best option here? I thought Vector had a built-in "zip" transform, but seems it only supports it in combination with string splitting (using split's field_names argument)

finally, it probably falls into the "bloat" department, but has there been any thought to something like a streaming HTTP POST src/sink or e.g. WebSocket src/sink? I could make HTTP sink's batch_size arbitrarily small, but I'm worried about side effects of doing that (not least, massive overhead introduced by HTTP headers and/or new TCP connections)

it seems odd to have this fast and slick transform binary that is forced to do many heavy roundtrips in the last hurdle :) I know there is plain socket available, but that is not something I want to expose on this network

1 reply

Kris Reeves

@myndzi

Hi there. Really cool project, excited to explore it :) I was reading the docs and was a little surprised to see that the http source is "best effort" while the splunk hec source is "at least once". It doesn't seem as though Splunk's HEC acknowledgement protocol is implemented, so I'm wondering what the difference is?

6 replies

(and on a similar topic, are there any plans to implement HEC acks?)

3 replies

Kris Reeves

@myndzi

Got one more question, related to content transformation. For things like referer links in web logs, we use Unilog to perform redaction of data before it hits disk. I can see that LUA stuff would maybe suffice here, or maybe some of the other transforms. However, with a big list of query string keys, we found significant performance problems which we were able to improve by using the aho-corasick algorithm to match the (fixed) list of query string values efficiently against the log data. It seems like there exists an implementation of this in LUA, but I'm not entirely sure what to expect for performance or managing non-trivial code. Any advice on what'd be most suitable here to replace that functionality?

2 replies

Luca Palmieri

@LukeMathWalker

Hi everyone!
I have been experimenting with Vector to ship structured logs from a Rust backend service - so far so good, I managed to get a setup that satisfies all my requirements.
A dilemma comes up when I need to deploy that bad boy as a Kubernetes pod. I have checked the docs, but I haven't found an answer (or a clear cut one): what is the recommended strategy to scrape logs from a container running in a k8s pod using vector?
Should vector run in a separate container on the same pod, sharing a volume, using the file source? What are the other available options?

5 replies

Ana Hobden

@Hoverbear

@LukeMathWalker Hey! :) I think using the docker source will work ok? Since this is a service you made you can just drop Vector right into the docker container

24 replies

Andrey Afoninsky

@afoninsky

hello

I want to implement docker proxy to connect Vector to MQTT (as a replacement of Kafka in event-based architecture)
which transport it's better to choose for communication between vector container and proxy container?
"vector source/sink" looks like a native solution, but "http source/sink" has at-least-once guarantee

Ana Hobden

@Hoverbear

So Vector <-> Vector may change in the future to include other APIs like timberio/vector#2003

Where HTTP would be passing just the logs

So if you want a stable API into your MQTT probably HTTP?

1 reply

Andrey Afoninsky

@afoninsky

true
on the other hand, I hope to replace this hack with native vector mqtt support later :)
plus, need to implement additional http logic: bulks, retries, etc...
but got it, thank you for the information

Ana Hobden

@Hoverbear

Ah yeah, gotcha. :) That'd be nice to have for sure

Alexandre NICOLAIE

@xunleii

Hi everyone :)

I've a question about the Loki sink. Is it a way to send a field (message for example) instead the complete event ? Because, inside Grafana, the Vector event is a JSON object and we can't currently get statistics inside the message field directly through the explorer. (And because we can generate labels from Vector, sending the full event is not really interesting).

1 reply

davewleblanc

@davewleblanc

Hi. :-) In your docs, you say "A config can have any number of transforms and it's entirely up to you how they are chained together." But there's no real example of how to do that. I have a pretty specific regex transform, but I'd like to match on more than one case, which would include another specific regex. But when running the tests fail because I have more than one.

1 reply

matrixbot

@matrixbot

@mike.cardwell:grepular.com The globbing in the file source, can you use multiple asterisks in multiple locations. E.g is this valid? include = ["/opt/nomad/data/alloc/*/alloc/logs/monitor.std*.*"]

1 reply

Slawomir Skowron

@szibis_twitter

Hi I like to send internal metrics from vector do datadog as metrics using datadog_metrics my config looks like this

[sources.internal_metrics]
  type = "internal_metrics"

[transforms.tags_internal_metrics]
  # General
  type = "add_tags" # required
  inputs = ["internal_metrics"] # required

  # Tags
  tags.hostname = "${VECTOR_HOSTNAME}"
  tags.role = "${VECTOR_ROLE}"
  tags.cluster = "${VECTOR_CLUSTER}"
  tags.env = "${VECTOR_ENV}"
  tags.region = "${VECTOR_REGION}"
  tags.project = "${VECTOR_PROJECT}"
  tags.hostgroup = "${VECTOR_HOSTGROUP}"

[sinks.internal_metrics_log]
  # General
  type = "console" # required
  inputs = ["tags_internal_metrics"] # required
  target = "stdout" # optional, default

  # Encoding
  encoding.codec = "json" # required
  encoding.timestamp_format = "rfc3339" # optional, default

[sinks.datadog_metrics_internal_metrics]
  # General
  type = "datadog_metrics" # required
  inputs = ["tags_internal_metrics"] # required
  api_key = “<“secret key> # required
  healthcheck = true # optional, default
  host = "https://app.datadoghq.com"
  namespace = "vector" # required

  # Batch
  batch.max_events = 20 # optional, default, events
  batch.timeout_secs = 1 # optional, default, seconds

tags are added and metrics show in logs in format that looks valid but Datadog sing taking 404 response

Apr 23 11:39:48 ip-172-18-99-67 vector[24857]: Apr 23 11:39:48.217 WARN sink{name=datadog_metrics_internal_metrics type=datadog_metrics}:request{request_id=1}: vector::sinks::util::retries: request is not retryable; dropping the request. reason=response status: 404 Not Found

19 replies

matrixbot

@matrixbot

@mike.cardwell:grepular.com > <@mike.cardwell:grepular.com> The globbing in the file source, can you use multiple asterisks in multiple locations. E.g is this valid? include = ["/opt/nomad/data/alloc/*/alloc/logs/monitor.std*.*"]

To answer my own question: yes. The problem I was having was that Vector could not see these files because the "alloc" dir was owned root:root with mode 0711, which meant the vector user couldn't get a directory listing, therefore the globbing failed. I feel like this should have been logged by Vector, but it wasn't.

1 reply

Rick Richardson

@rrichardson

For Kubernetes, the logging integration is awesome. Has anyone considered also leveraging the node-exporters in some way to collect all of the container and system metrics as well?

I think all that would be needed would be a scraper that acts as a vector source

A really nice-to-have would be a way to apply a filter at that point, to cull the data a bit and remove duplicates

Binary Logic

@binarylogic

[Mikhail Novichkin, Timber] We'll most likely provide integration for this via out Helm Chart os it's available via a single knob

matrixbot

@matrixbot

@mike.cardwell:grepular.com I don't know if this is a known issue, but there are no nightly rpms atm, just .debs: https://packages.timber.io/vector/nightly/latest/

1 reply

Slawomir Skowron

@szibis_twitter

Is anyone have problems with aws_ec2_metadata all I see in logs is

Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.004 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/placement/availability-zone
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.004 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/local-hostname
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.005 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/local-ipv4
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.005 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/mac
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.005 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:a5:31:79:0f:ac/subnet-id
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.006 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:a5:31:79:0f:ac/vpc-id
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.025 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/placement/availability-zone
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.025 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/placement/availability-zone
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.026 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/local-hostname
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.026 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/local-hostname
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.026 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/local-ipv4
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.026 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/local-ipv4
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.027 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/mac
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.027 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/mac
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.027 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:a5:31:79:0f:ac/subnet-id
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.027 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:a5:31:79:0f:ac/subnet-id
Apr 27 14:26:26 ip-10-105-195-187 vector[18040]: Apr 27 14:26:26.028 DEBUG aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Sending metadata request. uri=http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:a5:31:79:0f:ac/vpc-id

21 replies

gtie

@gtie

I'm not sure who is responsible for the release notes production/automation, but https://vector.dev/releases/0.9.0/ is pure joy

I'd go on a date with this page if I could ;)

amazing job!

Binary Logic

@binarylogic

[zach, Timber] Lol @Ben

Binary Logic

@binarylogic

@gtie ha ha, thanks, we worked hard on them :)

Alex

@Alexx-G

Hi everyone,
What's the recommended strategy on filtering out some logs? Is it acceptable creating a swimlane and leaving one flow not connected to a sink?

1 reply

Filippo Giunchedi

@filippog_gitlab

Hello, I'm trying out Vector and I'm wondering how to handle configurations with multiple transforms all needing to converge into a single sink. In other words, my use case is being able to add a transform and have the sink pick it up e.g. based on its name, without changing the sink's input list. Something like being able to say to the sink "use all transforms starting with 'blah' as inputs" without listing all input names explicitly, is this possible and/or planned? Thank you!

SvenMarquardt5772

@SvenMarquardt5772

i have a regex that doesn't match even though it matches in the syntax checker of rust.
the pattern is

^(?P<level>[\w\.]+) \[(?P<threadname>.*)\]: (?P<logger>[\w\.]*):(?P<linenumber>[\d\.]*) - (?P<message>.*).*E:(?P<exception>.*)?\n?(?P<stacktrace>(?s).*)?$

and the message is
INFO [Heartbeat]: SQSClientImpl:359 - Reset fffc3ffbcr Dev_SQS_Queue E:
Is there something wrong with my pattern?

10 replies

Ana Hobden

@Hoverbear

@filippog_gitlab While we don't have that feature right now, but I don't think that feature is out of the question! Can I invite you to open an issue?

Where communities thrive

People

Repo info

Activity