syslog-ng/syslog-ng

syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.

Balazs Scheidler

@bazsi

Are you sure you have kafka in your build?

Sanjay Patel

@San_j_ay_twitter

@bazsi I have 3.31

how do I verify if kafka is in the build?

Balazs Scheidler

@bazsi

Run syslog-ng --module-registry

Sanjay Patel

@San_j_ay_twitter

not in there

that is my question, how do I add kafka-c to my current install?

Balazs Scheidler

@bazsi

Which platform?

Robert Paschedag

@rpasche

Hi everyone....we are looking for alternatives to logstash and want to test syslog-ng. We are using a lot of filters and grok to - sometimes - create "nested objects" (within elasticsearch). Can someone show me a simple example, howto extract information from within a syslog message (${MESSAGE}) field, so we are able to "create" a new field? Example: Syslog message contains VM some_super_shiny_vm_name has been started. I want to add a new - nested - field like vm.name (JSON: {"vm": {"name": "my_super_shiny_vm_name"}}with the name of the VM ("my_super_shiny_vm_name"), so I could create a json object like

$(format-json message=${MESSAGE} vm.name=${my_new_extracted_field})

Thanks you for help.

6 replies

Joep van Diessen

@joepvand

Hello everyone, i currently have syslog-ng, but i want a way where a custom webinterface can see logs in realtime. does anyone know where to start with this?

2 replies

Sanjay Patel

@San_j_ay_twitter

I cant get syslog-ng to accept messages, when I do tcpdump I an see port open and messages coming in but when I run syslog-ng it does not accept the messages.

Sanjay Patel

@San_j_ay_twitter

I can see the port open but when I do syslog-ng -Fevd it shows no data

László Várady

@MrAnno

@San_j_ay_twitter Could you share the output of syslog-ng -Fevd?

Sanjay Patel

@San_j_ay_twitter

how do I send without flooding?

64 replies

Sanjay Patel

@San_j_ay_twitter

still stuck

tcpdump shows packets coming on udp but syslog-ng not picking them up

László Várady

@MrAnno

Hi,

If I understood your last reply correctly, netcat does not receive messages/packets either.
In case that's true, your issue is not really a syslog-ng configuration issue, it must be something environmental.

Sanjay Patel

@San_j_ay_twitter

@MrAnno it was a IPtable issue. Even with firewall off Ubuntu firewall rules apply. I added this to make it work. iptables -I INPUT -p udp -j ACCEPT. Adding it here incase someone else runs into this issue.

Robert Paschedag

@rpasche

Short question: Is the last pattern .*some kind of "workaround" within the new regexp-parserpattern so a message does not get discarded in case none of the previous pattern finds anything? Because I'm currently testing this new regexp-parser and added several patterns to extract information and noticed, that my message was never written to the destination. This seems to be caused by the fact, that none of my patterns really matched something. So is some kind of last pattern like .* the only chance I do not lose a message or are there flags that I can use to prevent messages to be discarded? Thank you for your help

16 replies

Robert Paschedag

@rpasche

Can you help me once more? I want to convert the syslog-ng TAGS into a JSON array, that should then be used within elasticsearch. Currently, the TAGS are set, but set a a static string (comma separated). I tried it with some rewrite but currently I cannot get it work. So I tried for example

rewrite {
  set("$(explode ',' ${TAGS})", value("tags"));
};

Balazs Scheidler

@bazsi

Just use the list type hint

1 reply

Fabien Wernli

@faxm0dem

rpasche: FWIW you could also use an Elasticsearch ingest pipeline : https://www.elastic.co/guide/en/elasticsearch/reference/7.15/split-processor.html

2 replies

Fabien Wernli

@faxm0dem

we like to do most in syslog-ng too, but when we have json upstream, we don't want syslog-ng to mess up its types, so we use ingest pipelines to parse the json in ES

just for the record: I understand your use-case

ann-lang

@ann-lang

Hello,
I am an assistant professor at Beihang University. Our team is doing a study about GSoC mentors, aiming to understand the motivations, challenges, strategies, and gains of GSoC mentors. To this end, we designed a questionnaire. We sincerely invite GSoC mentors to participate in this survey. Your feedback is very important for us.

Questionnaire link: https://forms.gle/rgAWwmrvrCb5XdAq9

If you are interested in this study, welcome to join our follow-up interview! Thank you very much!

Sincerely,

Xin Tan

2 replies

kiphackman

@kiphackman

hi guys im getting an issue on syslog-ng version 3.23.1 where logs are not written to the disk. I've spent a lot of time troubleshooting but had no luck. Would love some insight or assistance from the community if you guys may have any recommendations. Sorry for the trouble. And thank you so much

6 replies

Wojciech Adam Koszek

@wkoszek_gitlab

Imagine you're building a hobby project. You don't want to invest in Logstash'es and Kibanas. You love syslog-ng however. What's the easiest way to get some dashboard for logs?

To give you an example: modern tool to get web stats could be Grafana or something, but one can use Webalizer tool that has 1995 look, but does the job. I wonder if there's some 1995 tool that just works and gives me simple querability without all the maintenance

2 replies

Fabien Wernli

@faxm0dem

I'm guessing grafana/loki would be the most straightforward route nowadays

Peter Czanik

@czanik

if you do not mind closed source and do not need scalability in the long run, I liked SEQ, when I tested it a while ago: https://www.syslog-ng.com/community/b/blog/posts/creating-a-new-http--based-syslog-ng-destination-seq

1 reply

Fabien Wernli

@faxm0dem_twitter

Hi I'm getting a json parsing error that I don't understand: https://gist.github.com/faxm0dem/463d632d2f4031d8236f3294a04a98f6

I'm guessing it's the NaN it doesn't like

Fabien Wernli

@faxm0dem_twitter

yeah that was it

sorry for the noise

svestenik

@svestenik

Hello

syslog-ng 3 (3.35.1)

looks like cisco parser is not working

i have followed up the guide outlined in https://www.syslog-ng.com/community/b/blog/posts/parsing-cisco-logs-in-syslog-ng

and when I introduce syslog traffic from my network through the cisco parser, nothing comes out

Basic config elements look like this:

source s_ciscoudp { udp (ip("0.0.0.0") port(5140) flags(no-parse,store-raw-message)); };
destination d_raw { file("/var/log/raw" template("${RAWMSG}\n")); };
destination d_fromcisco { file("/etc/syslog-ng/fromcisco" template(t_jsonfile)); };

template t_jsonfile {
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs --key ISODATE)\n\n");
};

parser p_cisco {
cisco-parser();
};

log { source(s_ciscoudp); destination(d_raw); parser(p_cisco); destination(d_fromcisco); };

and i am getting the traffic into syslog-ng

destination d_raw works, raw messages get written

but nothing happens afterwards

Any ideas on how to find out whats wrong

Fabien Wernli

@faxm0dem

svestenik: hi

1 reply

you might have more clues when activating the internal() source

it will output internal logs of syslog-ng

if that doesn't give you any more clues, use the debug mode

syslog-ng -evdf /etc/syslog-ng/syslog-ng.conf

Gábor Nagy

@gaborznagy

Hi @svestenik What @faxm0dem suggested is a good idea: you need to know where is your message dropped.
Since you have a relatively simple log path with two destinations and a parser, I suspect the message is dropped in the p_cisco cisco-parser.
Cisco parser drops the message if it's format doesn't match a supported cisco format.
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.33/administration-guide/79#TOPIC-1663463

Ohh, sorry, you already figured that out...

Where communities thrive

People

Repo info

Activity