Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 08:30
    JeanMertz synchronize #7197
  • 07:08
    leebenson synchronize #7166
  • 05:39
    dependabot[bot] labeled #7422
  • 05:39
    dependabot[bot] review_requested #7422
  • 05:39
    dependabot[bot] review_requested #7422
  • 05:39
    dependabot[bot] review_request_removed #7422
  • 05:39
    dependabot[bot] opened #7422
  • 05:39
    dependabot[bot] labeled #7421
  • 05:39
    dependabot[bot] review_requested #7421
  • 05:39
    dependabot[bot] review_requested #7421
  • 05:39
    dependabot[bot] review_request_removed #7421
  • 05:39
    dependabot[bot] opened #7421
  • 05:39
    dependabot[bot] labeled #7420
  • 05:39
    dependabot[bot] review_requested #7420
  • 05:39
    dependabot[bot] review_requested #7420
  • 05:39
    dependabot[bot] review_requested #7420
  • 05:39
    dependabot[bot] review_request_removed #7420
  • 05:39
    dependabot[bot] opened #7420
  • 05:39
    dependabot[bot] labeled #7419
  • 05:38
    dependabot[bot] review_requested #7419
Francois Joulaud
@francoisj_gitlab
Another unrelated question. It seems that the Rosie transform was postponed. Does anyone here know the reason ?
1 reply
jogster
@jogster
Hi All, I have started using vector for monitoring metrics with statsd_source/prometheus_sink . Currently this works fine on the single machine with prometheus running on the same machine. Any advice from anyone how to set-up when setting up a local statsd and connecting to a remote prometheus_sink do I need to use a local tcp_source and remote prometheus_sink. I am not clear how to connect vector instances together...
5 replies
Thomas Silvestre
@thosil
Hi, I'd like to know if vector could be used as a "gateway" to connect separated kafka instances? So an event produced by instance A will be forwarded to instance B.
Thanks
1 reply
Frédéric Haziza
@silverdaz

Hi, I must be doing something wrong, but I'd like to collect my python logs, (multiple microservices running in docker containers) into a centralized place, a container itself.
For that I configured my python loggers to send UDP datagrams to a given location, and at that location, I have Vector running, with the following config:

[sources.my_source_id]
  type = "socket"
  address = "0.0.0.0:9000"
  mode = "udp"

[sinks.out]
inputs   = ["my_source_id"]
type     = "console"
encoding = "text"

However, I get the following error:

INFO vector: Log level "info" is enabled.
Jan 08 17:16:14.241  INFO vector: Loading config. path="/etc/vector/vector.toml"
Jan 08 17:16:14.246  ERROR vector: Configuration error: unknown variant `socket`, expected one of `docker`, `file`, `journald`, `kafka`, `splunk_hec`, `statsd`, `stdin`, `syslog`, `tcp`, `udp`, `vector` for key `sources.my_source_id.type`
I'm running the latest docker image (the alpine one).
What am I missing?
Frédéric Haziza
@silverdaz
Issue tracked on Github: timberio/vector#1491
And I got the answer ;)
Thanks
Spencer Dixon
@SpencerCDixon
I have an integration question that someone here may be able to help with. I have a native macOS app and I want to use Vector for log collection to send to S3 or CloudWatch. What would be the recommended source type? Right now I have my app set up to fork a new process with Vector running listening on stdin and then writing logs to that and it seems to work great but wanted to see if the authors had different recommendations?
Spencer Dixon
@SpencerCDixon
Also, won't I need AWS creds to send logs to S3/CloudWatch which I can't give to clients for security reasons.
Perhaps the recommended approach would be Vector Agent (client) ---> Vector server (service) --> fan out to S3/Cloudwatch and have creds on the Vector server/service?
Luke Steensen
@lukesteensen
hi @SpencerCDixon! stdin seems like a good fit for that situation as long as you're aware there can be backpressure and have a logger that handles it safely. if not, you could still use stdin but configure something like a disk buffer to avoid backpressure and shed load. other simple options would be logging to a file and configuring vector to tail it, or logging over tcp/udp. we recommend whichever works best for you.
and yes, we recommend a two-tiered approach for situations like that. the client-side config can be very simple and not need any credentials, while the upstream instance fans in all of the smaller streams and does the actual writes to storage.
Spencer Dixon
@SpencerCDixon
Awesome, thanks for the feedback Luke!
Cédric Da Fonseca
@Kwelity
Hi there,
I'm once more considering Vector to replace my current "log ingestion process".
All the app that we want to monitor run on a docker container. So currently, we use fluentbit as the "log driver" to parse / transform the logs.
I've looked at vector's docker source module and it seems to work quite differently than fluentbit.
In our case fluentbit itself run in a container, from what I understood, the plugin was installed directly inside of docker, docker then route the logs to a port... inside a container.
If I wanted to use vector I would have to install on the server itself, bare metal ?
Would it be possible to run a container with both vector and docker installed in it so that I don't require to install docker to the server ? I'm far from a docker expert, but that seems like it could potentially cause some sort of problems (to run a docker inside a container) ?
Maybe you have some experience with a similar flow ?
Cheers.
1 reply
Andrey Afoninsky
@afoninsky
hello guys, nice product you're doing
just wondering, do you have a scheduled date of next release? or expectations?
1 reply
Andrey Afoninsky
@afoninsky

a small feedback based on evaluation of vector in staging (close to production) on mambu.com

In general, I've found it useful and easy to build composable observabity architectures. I expect it will allow us to move from vendor-locks and be more flexible in solutions we are going to do.

I've splitted vector into small microservices, created helm chart and deployed them to kubernetes environment. It's easier to control them in that way and flexible enough. Every service performs a single operation: receives logs from AWS ELB, redirects logs to datadog, converts logs to metrics, expose metrics as prometheus collector, etc.... Also some of vectors acts as agent on external hosts, collecting information and sending it to common queue.

Things I miss, please inform if there is an issue or want me to create one:

  • ability to expose vector metrics and make them as a part of pipeline (found an issue about it so it's planned)
  • http endpoints which acts as kubernetes healtchecks, both readiness and liveness
  • javascript parser (found an issue, looks like a killer-feature to me as I will be able to move 99% of processing logic into vector)
  • kubernetes operator, in further I want to provide access for other teams so they will be able to create own pipelines using CRDs (will evaluate it later, does not look very complicated to create my own)
  • reading configs from yaml files, it's useful to generate configs based on template using jsonnet or helm (I think I've seen it planned)
2 replies
Andrey Afoninsky
@afoninsky
a question: if I use "log_to_metric" transform and later push to "statsd" sync - is it possible to specify metric timestamp? usecase: store events from http logs in past
7 replies
Andrey Afoninsky
@afoninsky
https://github.com/timberio/vector/blob/master/SECURITY.md
"That's why we apply widely accepted best practices when it comes to security"
is there any description which practices are applyed, and how? not an issue, but it would help to pass internal security check :)
Cory Kennedy-Darby
@ckdarby

Vector (Sink) -> Vector (Input)

Is there no TLS support?

Binary Logic
@binarylogic
[Lucio Franco, Timber] @ckdarby We don't currently have support for source side TLS. I've opened an issue related to this timberio/vector#1553
Kadir
@ktugan

Hi, quick question since I couldn't find an issue or any similar issue on the web. I am using the kinesis firehose sink but it seems to be unable to get the stream I defined.

Error:

Jan 22 16:29:36.082 ERROR vector::topology::builder: Healthcheck: Failed Reason: Stream names do not match, got other-independent-stream, expected my-log-stream

Config:

# Set global options
data_dir = "/var/lib/vector"

# Ingest data by tailing one or more files
[sources.server_queries]
  type         = "file"
  include      = ["/opt/server/queries.log.*"]
  ignore_older = 86400

[transforms.server_query_parser]
  type = "json_parser"
  inputs = ["server_queries"]
  drop_invalid = true

[sinks.kinesis]
  type = "aws_kinesis_firehose"
  inputs = ["server_query_parser"]
  region = "eu-west-1"
  stream_name = "my-log-stream"   <---------------
  encoding = "json"
  healthcheck = false

Would anybody know and can give hints what the cause might be?

2 replies
Kadir
@ktugan

Hi,

not sure if a bug or I do something wrong. Hopefully somebody can confirm.

The systemd vector service cannot be started because it looks like that the user+group is not created when installing the RPM:

Jan 23 12:04:56 ip-172-31-44-22.eu-west-1.compute.internal systemd[1]: Started Vector.
Jan 23 12:04:56 ip-172-31-44-22.eu-west-1.compute.internal systemd[1]: Starting Vector...
Jan 23 12:04:56 ip-172-31-44-22.eu-west-1.compute.internal systemd[1]: vector.service: main process exited, code=exited, status=217/USER
Jan 23 12:04:56 ip-172-31-44-22.eu-west-1.compute.internal systemd[1]: Unit vector.service entered failed state.
Jan 23 12:04:56 ip-172-31-44-22.eu-west-1.compute.internal systemd[1]: vector.service failed.

After I modified /usr/lib/systemd/system/vector.service and changed the users to root it worked:

8,9c8,9
< User=vector
< Group=vector
---
> User=root
> Group=root

The RPM was installed per docs on amazon linux 2 with following commands:

curl -O https://packages.timber.io/vector/0.7.X/vector-x86_64.rpm
sudo rpm -i vector-x86_64.rpm

I checked the issues but couldn't find a fitting one. Is the RPM supposed to create a new user+group?

1 reply
Andrey Afoninsky
@afoninsky
hi all, is it possible achieve pushing to Loki using http sync? https://github.com/grafana/loki/blob/master/docs/api.md#post-lokiapiv1push (I see an issue but who knows when it will be implemented :) timberio/vector#557 )
2 replies
Andrey Afoninsky
@afoninsky
is it possible to run vector as non-root? https://github.com/timberio/vector/blob/master/distribution/docker/alpine/Dockerfile
usecase example: able to run vector in kubernetes with pod security context enabled (runAsNonRoot=true)
3 replies
Andrey Afoninsky
@afoninsky
by the way, I’m maintaining own helm chart for internal purposes
here is a current snapshot for demonstration: https://github.com/afoninsky/vector-helm-adhoc (will delete in a few days)
just in case, maybe you will find something useful %)
1 reply
Andrey Afoninsky
@afoninsky
a question: is it possible to accumulate percentiles somehow on vector level based on raw events?
8 replies
gedkins
@gedkins
There's a couple of us here interested in developing http and TLS source support for Vector (timberio/vector#328 and https://github.com/timberio/vector/issues/1553). We have some code ready! Should we submit everything we have as one pull request or chop it up into pieces (e.g. one for http, one for https, one for documentation, one for the logplex source refactoring)?
2 replies
Binary Logic
@binarylogic
[Ana Hobden, Timber] :wave: Hi folks! I'm Ana and tend to go by hoverbear on your favourite services. I'm new around here, but I'll be working on Vector primarily for the forseeable future. I'm really excited to cooperate with you! Please get in touch (ana@timber.io) or ping me if you need any reviews, an extra set of eyes, or just want to explore some features or the code of Vector with me!
Ana Hobden
@Hoverbear
Hmm that didn't work right...
Sebastian YEPES
@syepes
Does
Does anyone know if there currently exists an influx line protocol source and sink available?
8 replies
Binary Logic
@binarylogic
[Ana Hobden, Timber] Yay! ScoopInstaller/Main#750
Andrey Afoninsky
@afoninsky
hello again
a questions regarded to "stream-based topology": https://vector.dev/docs/setup/deployment/topologies/
1) I see that vector has "kafka" source/sync, so it's possible to implement this topology... but it streams only "logs" events, what about "metrics"?
2) Do you plan to support less-expensive event-streams like "nats streaming server", or vendor-specific like "google pubsub"?
8 replies
3) not related to topology itself - what about 3d part of o11y: you've covered logs and events, are you planning to cover traces somehow?
2 replies
Ashley Jeffs
@Jeffail
hey winners, we're in the process of planning out some new goodies for our config spec, so far we have two proposals that aren't necessarily mutually exclusive but we'll likely only pick one to start with. If you have some time check out timberio/vector#1653 and timberio/vector#1679 and give a thumbs up/down on any proposals you like/dislike accordingly.
Andrey Afoninsky
@afoninsky
do you have a list of transforms, sources/syncs which will be introduced in 1.0.0? roadmap or something
1 reply
Ernie Turner
@ernieturner
Hi there :wave:
I've been looking into Vector to use as a logging service and I had a few high level questions that I was hoping to get some direction on, if that's not too much to ask.
9 replies
Jakub Bednář
@bednar

Hi All,

I am finishing implementation of influxdb sink and I have one question about integration tests. Where is a correct place to start InfluxDB for integration tests?
It’s ok if i add influxdb-docker image into docker-compose.yml and integration tests from influxdb_metrics.rs use it?

You can check progress of influxdb sink here: https://github.com/bonitoo-io/vector/blob/influxdb_metrics/src/sinks/influxdb_metrics.rs

Regards

cc @loony-bean @lukesteensen

3 replies
Andrey Afoninsky
@afoninsky

hello everyone
I have an issue with connecting to kafka from docker

current config:

[sources.kafka]
    type = "kafka"
    bootstrap_servers = "absolutly-sure-no-such-kafka-host"
    group_id = "elb-logs-consumer"
    topics = ["elb-logs"]

[sinks.out]
    inputs   = ["kafka"]
    type     = "console"
    encoding = "text"

args:

    Args:
      --config
      /config/vector.toml
      --require-healthy
      --verbose

looks like healtheck does not work:

Feb 07 13:34:42.017  INFO vector: Log level "debug" is enabled.
Feb 07 13:34:42.017  INFO vector: Loading config. path="/config/vector.toml"
Feb 07 13:34:42.018  INFO vector: Vector is starting. version="0.7.2" git_version="v0.7.2" released="Fri, 31 Jan 2020 20:33:36 +0000" arch="x86_64"
Feb 07 13:34:42.019  INFO vector::topology: Running healthchecks.
Feb 07 13:34:42.019  INFO vector::topology::builder: Healthcheck: Passed.
Feb 07 13:34:42.020  INFO vector::topology: All healthchecks passed.
Feb 07 13:34:42.020  INFO vector::topology: Starting source "kafka"
Feb 07 13:34:42.020  INFO vector::topology: Starting sink "out"

did I miss something?

4 replies
Andrey Afoninsky
@afoninsky
one more question :) I've found cloudwatch sync, but there are no information how to collect all these metric from cloudwatch... of course I will do something, but maybe there are some good practices you can recommend me based on the vector agent?
pierce
@wujiandong
Hi All I found that vector occasionally truncates logs. In my scenario, about 80,000 logs per minute, vector agent (tail)-> vector
1 reply
Feb 10 22:31:44.391 WARN transform{name=nginx_parser type=regex}: vector::transforms::regex_parser: Regex pattern failed to match. field="200.68.143.1
21 - - [10/Feb/2020:22:28:17 +0800] \"POST /[...]" rate_limit_secs=30
Feb 10 22:31:44.391 WARN transform{name=nginx_parser type=regex}: vector::transforms::regex_parser: "Regex pattern failed to match." is being rate lim
ited. rate_limit_secs=5
Grant Schofield
@schofield
Hi, I'm from Humio and was giving Vector a whirl today.. I was trying to read from from a Kafka source, after sending nginx logs with the Kafka sink... I get the following error, tried it in RUST_BACKTRACE=full mode but didn't add any additional context... I get this: Feb 15 00:54:42.530 ERROR source{name=kafka type=kafka}: vector::sources::kafka: Kafka returned error error=KafkaError (Message consumption error: NotImplemented (Local: Not implemented))

[sources.kafka]

REQUIRED

type = "kafka" # must be: "kafka"
bootstrap_servers = "broker1:9092,broker2:9092,broker3:9092"
group_id = "consumer-group" # example
topics = ["test-logs"] # example

is my config
this is an MSK cluster running 2.2
Binary Logic
@binarylogic
Hey @schofield ! Thanks for letting us know. I’ll file an issue and get someone on the team to take a look on Monday. It looks like something simple. My hope is that we can resolve it quickly.