vector/community

Discussions moved to Discord: https://discord.gg/jm97nzy

Ana Hobden

@Hoverbear

Hmm that didn't work right...

Sebastian YEPES

@syepes

Does

Does anyone know if there currently exists an influx line protocol source and sink available?

8 replies

Binary Logic

@binarylogic

[Ana Hobden, Timber] Yay! ScoopInstaller/Main#750

Andrey Afoninsky

@afoninsky

hello again
a questions regarded to "stream-based topology": https://vector.dev/docs/setup/deployment/topologies/
1) I see that vector has "kafka" source/sync, so it's possible to implement this topology... but it streams only "logs" events, what about "metrics"?
2) Do you plan to support less-expensive event-streams like "nats streaming server", or vendor-specific like "google pubsub"?

8 replies

3) not related to topology itself - what about 3d part of o11y: you've covered logs and events, are you planning to cover traces somehow?

2 replies

Ashley Jeffs

@Jeffail

hey winners, we're in the process of planning out some new goodies for our config spec, so far we have two proposals that aren't necessarily mutually exclusive but we'll likely only pick one to start with. If you have some time check out timberio/vector#1653 and timberio/vector#1679 and give a thumbs up/down on any proposals you like/dislike accordingly.

Andrey Afoninsky

@afoninsky

do you have a list of transforms, sources/syncs which will be introduced in 1.0.0? roadmap or something

1 reply

Ernie Turner

@ernieturner

Hi there :wave:
I've been looking into Vector to use as a logging service and I had a few high level questions that I was hoping to get some direction on, if that's not too much to ask.

9 replies

Jakub Bednář

@bednar

Hi All,

I am finishing implementation of influxdb sink and I have one question about integration tests. Where is a correct place to start InfluxDB for integration tests?
It’s ok if i add influxdb-docker image into docker-compose.yml and integration tests from influxdb_metrics.rs use it?

You can check progress of influxdb sink here: https://github.com/bonitoo-io/vector/blob/influxdb_metrics/src/sinks/influxdb_metrics.rs

Regards

cc @loony-bean @lukesteensen

3 replies

Andrey Afoninsky

@afoninsky

hello everyone
I have an issue with connecting to kafka from docker

current config:

[sources.kafka]
    type = "kafka"
    bootstrap_servers = "absolutly-sure-no-such-kafka-host"
    group_id = "elb-logs-consumer"
    topics = ["elb-logs"]

[sinks.out]
    inputs   = ["kafka"]
    type     = "console"
    encoding = "text"

args:

    Args:
      --config
      /config/vector.toml
      --require-healthy
      --verbose

looks like healtheck does not work:

Feb 07 13:34:42.017  INFO vector: Log level "debug" is enabled.
Feb 07 13:34:42.017  INFO vector: Loading config. path="/config/vector.toml"
Feb 07 13:34:42.018  INFO vector: Vector is starting. version="0.7.2" git_version="v0.7.2" released="Fri, 31 Jan 2020 20:33:36 +0000" arch="x86_64"
Feb 07 13:34:42.019  INFO vector::topology: Running healthchecks.
Feb 07 13:34:42.019  INFO vector::topology::builder: Healthcheck: Passed.
Feb 07 13:34:42.020  INFO vector::topology: All healthchecks passed.
Feb 07 13:34:42.020  INFO vector::topology: Starting source "kafka"
Feb 07 13:34:42.020  INFO vector::topology: Starting sink "out"

did I miss something?

4 replies

Andrey Afoninsky

@afoninsky

one more question :) I've found cloudwatch sync, but there are no information how to collect all these metric from cloudwatch... of course I will do something, but maybe there are some good practices you can recommend me based on the vector agent?

pierce

@wujiandong

Hi All I found that vector occasionally truncates logs. In my scenario, about 80,000 logs per minute, vector agent (tail)-> vector

1 reply

Feb 10 22:31:44.391 WARN transform{name=nginx_parser type=regex}: vector::transforms::regex_parser: Regex pattern failed to match. field="200.68.143.1
21 - - [10/Feb/2020:22:28:17 +0800] \"POST /[...]" rate_limit_secs=30
Feb 10 22:31:44.391 WARN transform{name=nginx_parser type=regex}: vector::transforms::regex_parser: "Regex pattern failed to match." is being rate lim
ited. rate_limit_secs=5

Grant Schofield

@schofield

Hi, I'm from Humio and was giving Vector a whirl today.. I was trying to read from from a Kafka source, after sending nginx logs with the Kafka sink... I get the following error, tried it in RUST_BACKTRACE=full mode but didn't add any additional context... I get this: Feb 15 00:54:42.530 ERROR source{name=kafka type=kafka}: vector::sources::kafka: Kafka returned error error=KafkaError (Message consumption error: NotImplemented (Local: Not implemented))

[sources.kafka]

REQUIRED

type = "kafka" # must be: "kafka"
bootstrap_servers = "broker1:9092,broker2:9092,broker3:9092"
group_id = "consumer-group" # example
topics = ["test-logs"] # example

is my config

this is an MSK cluster running 2.2

Binary Logic

@binarylogic

Hey @schofield ! Thanks for letting us know. I’ll file an issue and get someone on the team to take a look on Monday. It looks like something simple. My hope is that we can resolve it quickly.

We haven’t tested explicitly with Amazon’s Kafka service, so it might be that, but my assumption is that it shouldn’t matter.

Grant Schofield

@schofield

cool, also I got a strange error regarding humio, there aren't a lot of logs I'm shipping when using the file source but see Feb 15 01:16:26.584 TRACE sink{name=humio type=humio_logs}: tower_limit::rate::service: rate limit exceeded, disabling service

will check back Monday

Oh and the Kafka did work fine with our own kafka connect sink, and kafkacat works...

Binary Logic

@binarylogic

Thanks @schofield, we'll see what's going on with MSK. I've opened timberio/vector#1833 to look into it.

Regarding the rate limit error, you just need to raise https://vector.dev/docs/reference/sinks/humio_logs/#rate_limit_num . Let me know if we should raise the default there.

Grant Schofield

@schofield

thanks, missed that one, will give it a go

Andrey Afoninsky

@afoninsky

just noticed:
you've added "loki" sync as a part of the sprint
and it appeared in documentation which is good: https://vector.dev/docs/reference/sinks/loki/
the problem is - there is no information that it's a part of a nightly build, and not a part of latest stable (0.7.2) so it does not work

5 replies

leidruid

@leidruid_gitlab

hello, guys! I use vector as tcp > kafka transport.
some short time after the start, sink stucks, nothing happens, no suspicious messages in broker and vector logs (in debug mode too)

for a litte time before stuck, in vector log disappears sink messages (only source messages remains)

Has anyone encountered this problem?

4 replies

elbaro

@elbaro

Hello, I am routing glog file to elasticsearch.

[transforms.add_metadata] 
  type = "lua"
  inputs = ["src"]
  source = """
event['app'] = event['file']:match('/log/(.+)%.INFO')
"""

[sinks.proj_sink]
  type = "elasticsearch"
  inputs = ["add_metadata"]
  host = "http://1.2.3.4:5"
  index = "proj-{{host}}-{{app}}"

The above snippet does not work. The log shows nothing.

However

  index = "proj-{{host}}"

works and msgs have correct app field. Why can't I use {{app}} in index?

2 replies

Samuel Cormier-Iijima

@sciyoshi

Hi all, I'm not able to get the aws_ec2_metadata transform working. It seems to be connecting correctly to the metadata endpoint, but never makes a request. Here's my sample config:

data_dir = "/var/lib/vector"

[sources.stdin]
  type = "stdin"

[transforms.add_aws_metadata]
  type = "aws_ec2_metadata"
  inputs = ["stdin"]

[sinks.console]
  type = "console"
  inputs = ["add_aws_metadata"]
  encoding = "json"

And the debug logs:

Feb 19 21:59:30.069  INFO vector: Log level "info" is enabled.
Feb 19 21:59:30.069  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Feb 19 21:59:30.072  INFO vector: Vector is starting. version="0.8.0" git_version="v0.7.0-168-g841a8f8" released="Wed, 19 Feb 2020 11:14:46 +0000" arch="x86_64"
Feb 19 21:59:30.073 DEBUG trust_dns_resolver::async_resolver::background: trust-dns resolver running
Feb 19 21:59:30.073  INFO vector::topology: Running healthchecks.
Feb 19 21:59:30.073 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 21:59:30.074  INFO vector::topology: Starting source "stdin"
Feb 19 21:59:30.074 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))
Feb 19 21:59:30.075  INFO vector::topology::builder: Healthcheck: Passed.
Feb 19 21:59:30.076  INFO vector::topology: Starting transform "add_aws_metadata"
Feb 19 21:59:30.076  INFO source{name=stdin type=stdin}: vector::sources::stdin: Capturing STDIN
Feb 19 21:59:30.076  INFO vector::topology: Starting sink "console"
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 4194305
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 1
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 8388610
Feb 19 21:59:30.076 DEBUG tokio_reactor::background: starting background reactor
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 2
Feb 19 21:59:30.076 DEBUG tokio_reactor: adding I/O source: 12582915
Feb 19 21:59:30.076 DEBUG tokio_reactor::registration: scheduling Read for: 3
Feb 19 21:59:30.077 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG tokio_reactor::registration: scheduling Read for: 0
Feb 19 21:59:30.079 DEBUG hyper::proto::h1::io: flushed 102 bytes
test
{"host":"6792b4ce6d87","timestamp":"2020-02-19T21:59:37.278494604Z","message":"test"}

I'm running inside Docker using the nightly-alpine image. Any help/thoughts appreciated!

Actually, if I wait long enough I see

Feb 19 22:02:17.835 DEBUG hyper::proto::h1::conn: parse error (connection error: Connection reset by peer (os error 104)) with 0 bytes
Feb 19 22:02:17.835 DEBUG hyper::proto::h1::dispatch: read_head error: connection error: Connection reset by peer (os error 104)
Feb 19 22:02:17.835 DEBUG tokio_reactor: dropping I/O source: 0
Feb 19 22:02:17.836 ERROR aws_ec2_metadata: worker: vector::transforms::aws_ec2_metadata: Unable to fetch EC2 metadata; Retrying. error=connection error: Connection reset by peer (os error 104)
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connecting to 169.254.169.254:80
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor: adding I/O source: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: tokio_reactor::registration: scheduling Write for: 0
Feb 19 22:02:18.840 DEBUG aws_ec2_metadata: worker: hyper::client::connect::http: connected to Some(V4(169.254.169.254:80))

Maybe there's something else going on - although I'm able to curl the metadata endpoint from inside the container...

Lucio Franco

@LucioFranco

What type of instance are you on? and what does your curl command look like

Samuel Cormier-Iijima

@sciyoshi

thanks for the quick response @LucioFranco! it's a standard EC2 instance, m5.xlarge. here's the command I'm running:

admin@ip-172-20-98-28:~$ sudo docker run -it --entrypoint /bin/sh -e LOG=debug --rm --name vector -v $PWD/vector.toml:/etc/vector/vector.toml -v /var/lib/docker:/var/lib/docker -v /var/run/docker.sock:/var/run/docker.sock -v
$PWD/vector:/var/lib/vector -v /var/log/pods:/var/log/pods timberio/vector:nightly-alpine
/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/3) Installing nghttp2-libs (1.39.2-r0)
(2/3) Installing libcurl (7.66.0-r0)
(3/3) Installing curl (7.66.0-r0)
Executing busybox-1.30.1-r3.trigger
OK: 10 MiB in 19 packages
/ # curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "------------",
  "architecture" : "x86_64",
  "availabilityZone" : "ca-central-1a",
  "billingProducts" : null,
  "devpayProductCodes" : null,
  "marketplaceProductCodes" : null,
  "imageId" : "ami-0xxxxxx",
  "instanceId" : "i-0xxxxxx",
  "instanceType" : "m5.xlarge",
  "kernelId" : null,
  "pendingTime" : "2020-02-11T15:42:59Z",
  "privateIp" : "172.20.98.28",
  "ramdiskId" : null,
  "region" : "ca-central-1",
  "version" : "2017-09-30"
}/

Lucio Franco

@LucioFranco

ah looks like you're running vector within a container, that may be the reason

Samuel Cormier-Iijima

@sciyoshi

the curl command is also running from within the container

Lucio Franco

@LucioFranco

@sciyoshi can you try running the docker command with --net=host?

Samuel Cormier-Iijima

@sciyoshi

oh yup, that worked!! thank you :) not sure why curl would have been able to connect?

Lucio Franco

@LucioFranco

I would assume black magic :) glad that worked! let us know if you have any other issues.

Samuel Cormier-Iijima

@sciyoshi

I have another quick question - the json_parser transform seems to always remove the source field when drop_field is true. This seems inconsistent with the behavior of e.g. grok_parser, which only removes it when the parse succeeds. Is that behavior intentional?

Binary Logic

@binarylogic

Hey @sciyoshi , the behavior should be consistent across the two. I've opened timberio/vector#1861 to fix that.

Sebastian YEPES

@syepes

Small question, Is it currently possible to ingest (receive from UDP, TCP or file) metrics using the line protocol?

2 replies

Samuel Cormier-Iijima

@sciyoshi

@LucioFranco update on the original issue - I'm not able to use --net=host, but also it seems that it's only the /latest/api/token endpoint that is timing out from within a container. It seems that the API that should be used instead is the IMDS metadata - botocore updated due to this issue and you can see the changes here: boto/botocore#1895

7 replies

Andrey Afoninsky

@afoninsky

does vector have loggly support? haven't found any issues about it: https://github.com/timberio/vector/search?q=loggly&unscoped_q=loggly

1 reply

Aleksey Shirokih

@freeseacher

Hi! how can i transform something like that "file":"/var/log/mysystem/subsystem-component_name-07.log" to component_name ?

1 reply

Ana Hobden

@Hoverbear

Glad you got it!

Aleksey Shirokih

@freeseacher

as i can see there are type https://vector.dev/docs/about/data-model/metric/#aggregated_summary but how can i get it ? i am interested in prometheus summary of cause. there are some reference to timberio/vector#710 but i can't catch the point

Ana Hobden

@Hoverbear

@freeseacher if you're taking in logs and want to output metrics please try https://vector.dev/docs/reference/transforms/log_to_metric/

Aleksey Shirokih

@freeseacher

yes i am talking about metrics and already found log2metric but it does not help. type must must be one of: "counter" "gauge" "histogram" "set" but not quantile.

Samuel Cormier-Iijima

@sciyoshi

I am having issues with Docker log rotation using the default json-file logging driver - Vector stops picking up logs after the file is rotated

25 replies

Where communities thrive

People

Repo info

Activity

REQUIRED