by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jul 09 23:18
    Hoverbear synchronize #3003
  • Jul 09 23:17
    Hoverbear closed #1991
  • Jul 09 22:36
    bruceg synchronize #3005
  • Jul 09 21:09
    bruceg review_requested #3005
  • Jul 09 21:09
    bruceg review_requested #3005
  • Jul 09 21:09
    bruceg labeled #3005
  • Jul 09 21:09
    bruceg assigned #3005
  • Jul 09 21:09
    bruceg opened #3005
  • Jul 09 21:02
    Hoverbear closed #2969
  • Jul 09 21:00
    Hoverbear synchronize #2969
  • Jul 09 20:57
    Hoverbear edited #3004
  • Jul 09 20:56
    Hoverbear edited #3004
  • Jul 09 20:56
    Hoverbear review_requested #3004
  • Jul 09 20:56
    Hoverbear labeled #3004
  • Jul 09 20:56
    Hoverbear assigned #3004
  • Jul 09 20:56
    Hoverbear opened #3004
  • Jul 09 20:45
    Hoverbear closed #2987
  • Jul 09 20:45
    Hoverbear closed #2984
  • Jul 09 20:45
    Hoverbear closed #2983
  • Jul 09 20:44
    Hoverbear closed #2985
Jason Fehr
@jasonmfehr_twitter

@jasonmfehr_twitter @jasonmfehr_twitter

Architecture question -- I am leveraging the centralized topology (https://vector.dev/docs/setup/deployment/topologies/#centralized). Does Vector scale horizontally or do I need to stick with one Vector instance for the central service?

Just a follow-up from my post on June 15 at 9:40 (Pacific Time). Was wondering if anyone had thoughts or links to documentation I missed. Thanks!

Pastir
@Pastir

Hi all!
Please help me, I need to send logs to clickhouse without adding log_schema.
How can I send as is as?
For example, in the log file there is a line in json format:
{"key1": 2, "key2": 345}
I need to send the original string without any conversion:
{"key1": 2, "key2": 345}

I tried to use :)
encoding.except_fields = ["timestamp", "message", "host"]

Vlad Pedosyuk
@vpedosyuk
Hi all, are there any plans on a Kubernetes-based discovery for prometheus source?
4 replies
amitw-vmware
@amitw-vmware

I'm trying to read input in from a file. I can't figure out what i'm doing wrong. Here is my config:
```data_dir = "/Users/amitw/vector/data" # optional, no default

[sources.my_source_id]
type = "file" # required
include = ["/Users/amitw/vector/myLogFile.log"] # required
start_at_beginning = true # optional, default

[sinks.bar]
inputs = ["my_source_id"]
type = "console"
encoding.codec = "text"```

output when trying this on my mac
''' amitw@amitw-a01 vector % ls
data myLogFile.log vector.toml
amitw@amitw-a01 vector % ls /Users/amitw/vector/myLogFile.log
/Users/amitw/vector/myLogFile.log
amitw@amitw-a01 vector % vector --verbose --config /Users/amitw/vector/vector.toml
Jun 25 21:24:49.712 INFO vector: Log level "debug" is enabled.
Jun 25 21:24:49.712 INFO vector: Loading configs. path=["/Users/amitw/vector/vector.toml"]
Jun 25 21:24:49.715 INFO vector: Vector is starting. version="0.9.2" git_version="v0.9.2" released="Tue, 09 Jun 2020 22:01:29 +0000" arch="x86_64"
Jun 25 21:24:49.716 INFO vector::topology: Running healthchecks.
Jun 25 21:24:49.716 INFO vector::topology: Starting source "my_source_id"
Jun 25 21:24:49.716 INFO vector::topology: Starting sink "bar"
Jun 25 21:24:49.716 INFO vector::topology::builder: Healthcheck: Passed.
Jun 25 21:24:49.716 INFO source{name=my_source_id type=file}: vector::sources::file: Starting file server. include=["/Users/amitw/vector/amit.txt"] exclude=[]'''
amitw@amitw-a01 vector % tail -f /Users/amitw/vector/myLogFile.log
G
H
I
J
K
L
M
N
O
P
amitw-vmware
@amitw-vmware
It looks like my file size was too small. If i added more data to my myLog.log file it worked!
amitw-vmware
@amitw-vmware
I'm finding it pretty difficult to exclude lines with any words in a set since regex support does not negative look-ahead. Any plans to support this better in the future?
1 reply
leader32
@leader32
Hi. "sources.file" grabs json logs from file, then I try to apply "transform.rename_fields", it says there's no that field. What can I do to solve my problem?
Jakob Sievers
@cannedprimates
Hi, I'm trying out vector, and it appears to be working as expected (events flow from my sources to my sinks) but almost all events seem to be dropped (only about 10% of the events my sources produce show up in my sinks)
im running with -vv and there are no errors in the logs (though i did see some messages about rate limiting)
i did notice that vector was using 100% cpu though, and when sendign it a sigterm it fails to shut down properly
any suggestions for how i go about debugging this?
5 replies
pepoon
@pepoon
Is it possible for a an aws todo a health check on vector app? thanks.
5 replies
leader32
@leader32
How to convert the ISO 8601 timestamp to standard timestamp? Tried to use "transforms.coercer" with no result.
pepoon
@pepoon
server->tcp_connection->Vector.dev->s3. Vector.dev only sinks out to s3 when i close the tcp connection.
Any idea why..
batch.max_size = 1000 # (test)
batch.timeout_secs = 10 # (test)
are getting ignored?
Thanks.
andrew4fr
@andrew4fr

Hi all

How can I split "remove_secret" output so events with docker label "file_log=true" go to [sinks.http] and events with label "elastic_log=true" go to [sinks.elastic]?

[sources.in]
  type = "docker"
  include_labels = ["file_log=true", "elastic_log=true"]

[transforms.json]
  type = "json_parser"
  inputs = ["in"]
  drop_invalid = true

[transforms.remove_secrets]
  type = "lua"
  inputs = ["json"]
  version = "2"

  hooks.process = """
  function (event, emit)
    emit(event)
  end
  """

[sinks.http]
  type = "http"
  inputs = ["to_file"]
  uri = "10.1.1.126:80"
  encoding.codec = "ndjson"

[sinks.elastic]
  type = "elasticsearch"
  inputs = ["to_elastic"]
  host = "10.1.1.127:9000"
  index = "request-%Y-%m-%d"
5 replies
Keery Nie
@windmgc

Hi I wonder if there's a missing flush after this line: https://github.com/timberio/vector/blob/02c30220d6a2a34425b33fc25490a3520d748282/src/sinks/file/mod.rs#L228

I found that file sink did not sink message in time, which may cause message delay and lost

3 replies
heinzlea-eca
@heinzlea-eca
Hi all ! I use vector for 2 weeks now and it's a very interesting project ! i try to read a lot of CSV files and want to write them in InfluxDB. Everything works fine but some files are not read. Indeed it says that some files have same fingerprint and so only read the newest one. Howerver my files are different, they are in separate folders and are named file_20200416_052357_074824.csv and file_20200423_052030.csv for example. They contain same columns but at different timestamps and are so different. Is there a way to force vector to read all the files ?
9 replies
Pratyush Singhal
@psinghal20
Hi, I am facing an issue while testing my changes. I used the Nix setup mentioned in the contributing guide, but the build is failing because of a missing library sasl2which is required in the linking process. I do have the libsasl 2.1 on my machine outside the Nix environment. Can someone help me out with this? Thanks!
Vlad Pedosyuk
@vpedosyuk
Hi, does elasticsearch sink support a so-called backpressure-sensitive protocol like Filebeat? As far as I understand, it doesn't...?
Felipe Passos
@SharksT
Hello, i have 2 apis and a websocket in a server, a dashboard in another. What is the best way of use of vector to aggregate those data and put that data on prometheus/grafana?
Jakob Sievers
@cannedprimates
hi, I believe I've tracked my issue (cf above) to the aws_s3 sink: vector successfully reads all lines from the source file but then the sink hangs and vector starts using 100% cpu. there are zero 'sending item' or 'submit successful' lines in the log (-vv). any way to trace whats going on in the sink or do i have to put in some print statements and recompile?
3 replies
Mark Klass
@ChristianKlass
Hi, is it possible for Vector to receive and log SNMP traps? I am considering Vector to see if it can fit our purposes.
andrew4fr
@andrew4fr
Knock, knock. Is there anybody out there?
Ana Hobden
@Hoverbear
Hey folks, sorry -- We've been bad about getting to this this week. Regional holidays combined with some folks moving (houses) has left us a bit shorthanded. I'm trying to go through the backlog
@ChristianKlass I don't think anyone has tried that yet. We might need to add support. I suggest opening an issue.
@SharksT You probably are writing logs already somewhere? Can you have vector read those to start?
@psinghal20 Yeah -- We're in the process of reverting this change. We recommend you remove it. Sorry about that. It was a well intentioned experiment.
Felipe Passos
@SharksT
Yes, I’m writing the logs on /var/etc/logs
Ana Hobden
@Hoverbear
@vpedosyuk Our ES sink should backoff on failure, but in terms of e2e backpressure that's kind of in progress. You might be able to get some more info from @lukesteensen if you want
1 reply
@SharksT You can grab them with https://vector.dev/docs/reference/sources/file/ to start
@leader32 What is a "standard timestamp"?
Francesco Palazzini
@palazzini.francesco_gitlab
hello, do you think it would make sense to have an email sink?
use case: you might want to get notified via mail if some logs are produced
Ana Hobden
@Hoverbear
@palazzini.francesco_gitlab That sounds like something you should be doing in prometheus via alert manager etc?
You can push to prometheus with Vector, and turn logs into metrics via our log_to_metric transform
Francesco Palazzini
@palazzini.francesco_gitlab
@Hoverbear thanks for pointing that out!
in my case I do not use prometheus and would like to have that as a Vector sink.
I might implement it on my fork. I was asking because if that feature might be needed I could do a pr
Ana Hobden
@Hoverbear
@vpedosyuk (@lukesteensen) said in a thread that backpressure is handled. Gitter won't show you this on some platforms though)
@palazzini.francesco_gitlab Could I invite you to open an issue and spec out what you'd do in a bit of detail? Maybe we can merge it to main branch!
Luke Steensen
@lukesteensen
@vpedosyuk could you elaborate on what you mean by supporting a protocol? the elasticsearch sink uses the elasticsearch HTTP API and backpressure is handled by limiting the number of concurrent requests, rate limits, etc. that backpressure by default will flow all the back back to connected sources
Vlad Pedosyuk
@vpedosyuk
@Hoverbear @lukesteensen thanks for the answer. Maybe that's just my interpretation but when I read the Filebeat docs, it says it supports "a backpressure sensitive protocol". I thought: "cool! some sophisticated elastic-specific protocol". While other tools like Vector or Fluentbit "just" retry on a request failure. However, I've read more docs and it seems Filebeat works similarly.
Paul Barker
@pbthemaverik_twitter
Hi, I am trying vector after hitting some issues with fluentd to S3, where the syslog format to S3 was in JSON, and I need RAW output in RFC3164, so no processing of the inbound message?
venkatvghub
@venkatvghub_gitlab
Hi, i am trying to use vector to transform json records from kafka and write them back to another kafka topic. So, my parsing logic requires me to split a list of maps into a flattened map. For instance, i have something like this: {'tags':[{'key1':'value1'}, {'key2':'value2'}....} to {'tags': {'key1':'value1', 'key2':'value2'}....}. How do i go about achieving this?
Rick Richardson
@rrichardson
Is the branch here timberio/vector#2653 the best source to try the kubernetes_logs source?
Rick Richardson
@rrichardson
I just built the the latest from the PR branch for k8s. However, I get thread 'main' panicked at 'capacity overflow', src/liballoc/raw_vec.rs:622:5
given the nature of the issue, it is hard to google for :)
should I explicitly set a buffer size?
Rick Richardson
@rrichardson
It occurs here: 16: <vector::sinks::loki::LokiConfig as vector::topology::config::SinkConfig>::build