syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
Short question: Is the last pattern
.*some kind of "workaround" within the new regexp-parserpattern so a message does not get discarded in case none of the previous pattern finds anything? Because I'm currently testing this new regexp-parser and added several patterns to extract information and noticed, that my message was never written to the destination. This seems to be caused by the fact, that none of my patterns really matched something. So is some kind of last pattern like .* the only chance I do not lose a message or are there flags that I can use to prevent messages to be discarded? Thank you for your help
16 replies
Can you help me once more? I want to convert the syslog-ng
TAGS into a JSON array, that should then be used within elasticsearch. Currently, the TAGS are set, but set a a static string (comma separated). I tried it with some rewrite but currently I cannot get it work. So I tried for examplerewrite {
set("$(explode ',' ${TAGS})", value("tags"));
};
rpasche: FWIW you could also use an Elasticsearch ingest pipeline : https://www.elastic.co/guide/en/elasticsearch/reference/7.15/split-processor.html
2 replies
just for the record: I understand your use-case
Hello,
I am an assistant professor at Beihang University. Our team is doing a study about GSoC mentors, aiming to understand the motivations, challenges, strategies, and gains of GSoC mentors. To this end, we designed a questionnaire. We sincerely invite GSoC mentors to participate in this survey. Your feedback is very important for us.
Questionnaire link: https://forms.gle/rgAWwmrvrCb5XdAq9
If you are interested in this study, welcome to join our follow-up interview! Thank you very much!
Sincerely,
Xin Tan
2 replies
hi guys im getting an issue on syslog-ng version 3.23.1 where logs are not written to the disk. I've spent a lot of time troubleshooting but had no luck. Would love some insight or assistance from the community if you guys may have any recommendations. Sorry for the trouble. And thank you so much
6 replies
To give you an example: modern tool to get web stats could be Grafana or something, but one can use Webalizer tool that has 1995 look, but does the job. I wonder if there's some 1995 tool that just works and gives me simple querability without all the maintenance
2 replies
if you do not mind closed source and do not need scalability in the long run, I liked SEQ, when I tested it a while ago: https://www.syslog-ng.com/community/b/blog/posts/creating-a-new-http--based-syslog-ng-destination-seq
1 reply
Hi I'm getting a json parsing error that I don't understand: https://gist.github.com/faxm0dem/463d632d2f4031d8236f3294a04a98f6
I'm guessing it's the
NaN it doesn't like
sorry for the noise
syslog-ng 3 (3.35.1)
looks like cisco parser is not working
i have followed up the guide outlined in https://www.syslog-ng.com/community/b/blog/posts/parsing-cisco-logs-in-syslog-ng
and when I introduce syslog traffic from my network through the cisco parser, nothing comes out
Basic config elements look like this:
source s_ciscoudp { udp (ip("0.0.0.0") port(5140) flags(no-parse,store-raw-message)); };
destination d_raw { file("/var/log/raw" template("${RAWMSG}\n")); };
destination d_fromcisco { file("/etc/syslog-ng/fromcisco" template(t_jsonfile)); };
template t_jsonfile {
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs --key ISODATE)\n\n");
};
parser p_cisco {
cisco-parser();
};
log { source(s_ciscoudp); destination(d_raw); parser(p_cisco); destination(d_fromcisco); };
and i am getting the traffic into syslog-ng
but nothing happens afterwards
Any ideas on how to find out whats wrong
you might have more clues when activating the internal() source
it will output internal logs of syslog-ng
if that doesn't give you any more clues, use the debug mode
syslog-ng -evdf /etc/syslog-ng/syslog-ng.conf
Hi @svestenik What @faxm0dem suggested is a good idea: you need to know where is your message dropped.
Since you have a relatively simple log path with two destinations and a parser, I suspect the message is dropped in the p_cisco cisco-parser.
Cisco parser drops the message if it's format doesn't match a supported cisco format.
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.33/administration-guide/79#TOPIC-1663463
Since you have a relatively simple log path with two destinations and a parser, I suspect the message is dropped in the p_cisco cisco-parser.
Cisco parser drops the message if it's format doesn't match a supported cisco format.
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.33/administration-guide/79#TOPIC-1663463
Ohh, sorry, you already figured that out...
cisco-parser supports the following format:
<pri>(sequence: )?(origin-id: )?(timestamp? timezone?: )?%msg
Hello @dvlsatya !
We are more than happy to receive contributions from the community! :)
At first I suggest to compile syslog-ng.
You have several options: you can use our docker-based infrastructure and then you don't have to setup a build environment on your computer.
Here is a short description about the usage the docker-based build system (dbld):
https://github.com/syslog-ng/syslog-ng/tree/master/dbld#hacking-on-syslog-ng-itself
What kind of contribution do you have in mind?
We are more than happy to receive contributions from the community! :)
At first I suggest to compile syslog-ng.
You have several options: you can use our docker-based infrastructure and then you don't have to setup a build environment on your computer.
Here is a short description about the usage the docker-based build system (dbld):
https://github.com/syslog-ng/syslog-ng/tree/master/dbld#hacking-on-syslog-ng-itself
What kind of contribution do you have in mind?
I just updated to debian/bullseye, and the latest syslog-ng-master says
`Required bison not found /home/fwernli/git/syslog-ng/build/lib/rewrite/rewrite-expr-grammar.y
I've got `bison (GNU Bison) 3.7.5
ah I see the warning now in the configure script that I need at least 3.7.6
back to yak shaving then
I have created a new syslog configuration file in /opt/syslog-ng/etc/conf.d.
I have created one source and destination file inside script /var/log/myfile.txt
For some reason i have changed the destination file to myfile1.txt inside the configuration and reloaded the config file.
Though i have deleted myfile.txt, still is it there and updated with the same data as myfile1 has. I have deleted myfile.txt multiple times but still creating. Could someone please help on this?
I have created one source and destination file inside script /var/log/myfile.txt
For some reason i have changed the destination file to myfile1.txt inside the configuration and reloaded the config file.
Though i have deleted myfile.txt, still is it there and updated with the same data as myfile1 has. I have deleted myfile.txt multiple times but still creating. Could someone please help on this?
Hi @deepulikesbru!
That sounds strange. Syslog-ng writes to the new file after the reload.
Can you share with us the actual config, please?
Does the old file (myfile.txt) have the same content like the new myfile1 file after it's been re-created? Does it have the old myfile.txt contents as well?
That sounds strange. Syslog-ng writes to the new file after the reload.
Can you share with us the actual config, please?
Does the old file (myfile.txt) have the same content like the new myfile1 file after it's been re-created? Does it have the old myfile.txt contents as well?
@gaborznagy I removed the config file and created with different file names again inside the config file with mytext2.txt and deleted myfile.txt and myfile1.txt before reloading. I did see the files myfile.txt and myfile1.txt again after some time. Why these files are creating again after sometime though i have deleted the files? What could be the issue? Does syslog has internally anything to clear the buffer ??