Aug 10 21:30:21.451 INFO vector::sources::docker: Started listening logs on docker container id=7d704b317c21a893a26de131a0495e08ef39ee5144e0a743e23a6027c85316e2
Aug 10 21:30:21.454 TRACE vector::sources::docker: Received one event. event=Log(LogEvent { fields: {"container_created_at": Timestamp(2020-08-11T01:02:56.926028916Z), "container_id": Bytes(b"7d704b317c21a893a26de131a0495e08ef39ee5144e0a743e23a6027c85316e2"), "container_name": Bytes(b"mynginx13"), "image": Bytes(b"nginx"), "label": Map({"maintainer": Bytes(b"NGINX Docker Maintainers <docker-maint@nginx.com>")}), "message": Bytes(b"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"), "source_type": Bytes(b"docker"), "stream": Bytes(b"stdout"), "timestamp": Timestamp(2020-08-11T01:30:21.441879555Z)} })
Aug 10 21:30:21.454 WARN sink{name=cw_log type=aws_cloudwatch_logs}: vector::sinks::aws_cloudwatch_logs: keys in stream template do not exist on the event; dropping event. missing_keys=[Atom('host' type=inline)] rate_limit_secs=30
Aug 10 21:30:21.454 TRACE vector::sources::docker: Received one event. event=Log(LogEvent { fields: {"container_created_at": Timestamp(2020-08-11T01:02:56.926028916Z), "container_id": Bytes(b"7d704b317c21a893a26de131a0495e08ef39ee5144e0a743e23a6027c85316e2"), "container_name": Bytes(b"mynginx13"), "image": Bytes(b"nginx"), "label": Map({"maintainer": Bytes(b"NGINX Docker Maintainers <docker-maint@nginx.com>")}), "message": Bytes(b"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/"), "source_type": Bytes(b"docker"), "stream": Bytes(b"stdout"), "timestamp": Timestamp(2020-08-11T01:30:21.441906500Z)} })
Aug 10 21:30:21.454 WARN sink{name=cw_log type=aws_cloudwatch_logs}: vector::sinks::aws_cloudwatch_logs: "keys in stream template do not exist on the event; dropping event." is being rate limited. rate_limit_secs=5
Hey all :) When using the Loki
sink, I am continuously getting entry out of order
errors for my logs. Does anyone have any experience?
I (think) the problem is that because we deploy vector as a daemonset in our clusters (currently at 4 pods) it's a decentralised deployment, making things tricky for loki to handle. But we thought adding a unique label to each instance would solve the problem.
I added a unique label for each instance withlabels.vector_instance = "${HOSTNAME}"
but I'm still getting the same error. I can verify the label exists by inspecting the logs in grafana because some are coming through, but most are being rejected!
I've also tried:
labels.vector_instance = {{ host }}
request.in_flight_limit = 1
request.rate_limit_num = 1
Hey there
I got to know vector a few months ago and have been using it personally for all the non-profit projects I work on.
Now I would like to bring it's awesome performance to the team I work at currently but at that scope, it's not anymore just a small thing.
Our current setup consists of logstash scaling up or down dynamically depending on the load and all the apps pushing their filebeat, packetbeat, metricbeat, etc. data (elastic stack beats) to it. As I would like to transition to vector my first attempt would be bringing over vector to replace logstash to have all the current setup elastic stack compliant but benefit from the performance of vector.
I again read through some parts of your docs and found some stuff I'm currently concerned about.
One would be the at least delivered once guarantee https://vector.dev/docs/about/guarantees/#at-least-once
You state that you should specify some kind of path to have events queued up until they can be delivered once the destination is available again. A nice feature but that means my container has to be stateful?
I thought about having vector reading from a kafka topic but that raised also some questions. I read part of the kafka source code but couldn't find a part where the instances talk to each other. Does this mean each instance would consume the event? Is vector cloud native?
I may misunderstand some scaling vector but I couldn't find a part in the docs explaining those situations.
Is there a way to send internal_metrics
to statsd
sink via vector. I tried with it with this config
https://gist.github.com/perfectayush/3aaec7fd63439be69f07440f72b80ca6
But on when listening via netcat on 8125, all I am getting is the namespace, not the metrics.
Hi All,
Hope you can help me, I'm trying to stream data from Kafka to GCS.
The requirement is to create an object for each event from Kafka and the object name is compound from content in the event.
The first question, is it possible not to use the batch option? (or batch 1)
Second, I think I saw it possible to reference all fields can I use conversion and split of date from one field?
third, If I'm reading from Kafka can I skip disk buffer and still achieve at least one?
Thank you all!
Aug 23 02:28:47.114 ERROR sink{name=clickhouse-apilog type=clickhouse}:request{request_id=212}: vector::sinks::util::sink: Response wasn't successful. response=Response { status: 400, version: HTTP/1.1, h
eaders: {"date": "Sun, 23 Aug 2020 02:28:47 GMT", "connection": "Keep-Alive", "content-type": "text/tab-separated-values; charset=UTF-8", "x-clickhouse-server-display-name": "master-01", "transfer-encodin
g": "chunked", "x-clickhouse-query-id": "1188cca8-94ef-4b63-b3c9-19c7771ee72b", "x-clickhouse-format": "TabSeparated", "x-clickhouse-timezone": "UTC", "x-clickhouse-exception-code": "26", "keep-alive": "t
imeout=3", "x-clickhouse-summary": "{\"read_rows\":\"0\",\"read_bytes\":\"0\",\"written_rows\":\"0\",\"written_bytes\":\"0\",\"total_rows_to_read\":\"0\"}"}, body: b"Code: 26, e.displayText() = DB::Except
ion: Cannot parse JSON string: expected opening quote: (while read the value of key consumer.created_at): (at row 19)\n (version 20.6.3.28 (official build))\n” }
@db2jlu_twitter I'm not super familiar with Clickhouse, but there is an open issue for metrics support: timberio/vector#3435 . It may just not be implemented yet.
Looking at that though, are you sure that's the reason? It seems like it might be a mismatch in the schema or datatypes in clickhouse or, possibly, that vector is sending invalid JSON
I just posted a blog about Vector: https://www.splunk.com/en_us/blog/it/meet-the-fastest-forwarder-on-the-net.html
huh...Splunk pulled the article?
transforms:
"liran-demo-logs":
type: filter
inputs: ["kubernetes_logs"]
rawConfig: |
[transforms.liran-demo-logs.condition]
"kubernetes.pod_labels.component.eq" = "app=liran-demo"
"stream.eq" = "stdout"
sinks:
console:
type: "console"
inputs: ["liran-demo-logs"]
taget: "stdout"
rawConfig: |
# Encoding
encoding.codec = "json" # required
hello
I have a lot of spam messages after installing helm chart "vector-0.11.0-nightly-2020-08-24":
Aug 25 13:34:06.533 WARN source{name=kubernetes_logs type=kubernetes_logs}: vector::internal_events::kubernetes_logs: failed to annotate event with pod metadata event=Log(LogEvent { fields: {"file": Bytes(b"/var/log/pods/vector_cluster-logs-chf8d_290b7ab5-9752-49f1-81d7-cc9a51483c4d/vector/2.log"), "message": Bytes(b"{\"log\":\"Aug 25 13:19:17.029 INFO source{name=kubernetes type=kubernetes}:file_server: file_source::file_server: More than one file has same fingerprint. path=\\\"/var/log/pods/jaeger_jaeger-cassandra-2_3d357498-7fd7-448e-a0d7-54b8922b0050/jaeger-cassandra/6.log\\\" old_path=\\\"/var/log/pods/jaeger_jaeger-cassandra-2_3d357498-7fd7-448e-a0d7-54b8922b0050/jaeger-cassandra/5.log\\\"\\n\",\"stream\":\"stdout\",\"time\":\"2020-08-25T13:19:17.02974474Z\"}"), "source_type": Bytes(b"kubernetes_logs"), "timestamp": Timestamp(2020-08-25T13:34:06.533091773Z)} })
config:
kubernetesLogsSource:
enabled: true
sourceId: kubernetes_logs
env:
- name: LOGGLY_TOKEN
value: ****-****-****-****-****
sinks:
# console:
# type: console
# inputs: ["kubernetes_logs"]
# rawConfig: |
# encoding.codec = "json"
loggly:
type: http
inputs: ["kubernetes_logs"]
rawConfig: |
uri = "https://logs-01.loggly.com/bulk/${LOGGLY_TOKEN}/tag/olly,dev,k8s/"
batch.max_size = 50000
encoding.codec = "ndjson"
should I create an issue or it's already known and/or fixed? thanks
[sources.access-raw]
# General
type = "file"
ignore_older = 300
include = ["/var/log/od/access_*.log"]
start_at_beginning = false
oldest_first = true
fingerprinting.strategy = "checksum"
fingerprinting.ignored_header_bytes = 2048
fingerprinting.fingerprint_bytes = 4096
Aug 25 14:39:14 vm8857 vector: Aug 25 14:39:14.117 ERROR source{name=access-raw type=file}:file_server: file_source::file_server: Error reading file for fingerprinting err=Too many open files (os error 24) file="/var/log/od/access_2020-02-24_13-53-24_pid_2074.log"
I could change max_open_files, which is limited to 1024 for the vector user, but it seems odd to have to do such a thing when only one log file at a time is being written.
[sinks.loki]
# General
type = "loki" # required
inputs = ["cleaned_traefik_logs"]
endpoint = "http://loki:3100" # required
healthcheck = true # optional, default
# Encoding
encoding.codec = "json" # optional, default
# Labels
labels.key = "value" # I'm not sure what this does
labels.key = "{{ event_field }}" # nor this