syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
I don't really understand the question. I'd try something like this (not syntax checked, just from the top-of-my-head):
log {
source(whatever);
parser { db-parser(file('ironport.xml')); };
log {
filter { <match only iron port logs...>; };
if ('${CID}' ne '') {
parser { grouping-by(<CID aggregation options> inject-mode(passthrough)); };
};
if ('${MID}' ne '') {
parser { grouping-by(<MID aggregation options> inject-mode(passthrough)); };
};
if (<match aggregated message from either CID or MID based aggregation>) {
parser { grouping-by(<MID + CID result aggregation> inject-mode(passthrough)); };
};
filter { <match only aggregated logs emitted by the last grouping-by()>); };
};
destination(whatever);
};
the messages flow through the sequence one by one.
1) The source() statement in the front pulls in any messages received by the specific source.
2) the db-parser() right next would extract ironport name-value pairs from the messages
3) the embedded log {} statement just encapsulates a set of operations
4) the first filter embedded in the log {} statement would drop anything but ironport (I am not sore if the db-parser() field extraction sets a specific field or not, but that's certainly doable based on some criteria)
5) the first if() checks if $CID is set, if it is, a grouping-by() parser is executed, which should in turn emit an aggregated result of all messages where CID was set.
6) the second if() checks if $MID is set, similarly to the first, it would need to aggregate the $MID related messages into a result
7) the last if() would match messages that either the first or the second grouping-by() emitted and runs a 3rd grouping-by()
8) this third grouping-by() is our end result
9) the filter as the last statement within our log {} would drop anything but the aggregated result
10) the destination only receives the output of the entire log {} statement, e.g. the result of the 3rd aggregation
1) The source() statement in the front pulls in any messages received by the specific source.
2) the db-parser() right next would extract ironport name-value pairs from the messages
3) the embedded log {} statement just encapsulates a set of operations
4) the first filter embedded in the log {} statement would drop anything but ironport (I am not sore if the db-parser() field extraction sets a specific field or not, but that's certainly doable based on some criteria)
5) the first if() checks if $CID is set, if it is, a grouping-by() parser is executed, which should in turn emit an aggregated result of all messages where CID was set.
6) the second if() checks if $MID is set, similarly to the first, it would need to aggregate the $MID related messages into a result
7) the last if() would match messages that either the first or the second grouping-by() emitted and runs a 3rd grouping-by()
8) this third grouping-by() is our end result
9) the filter as the last statement within our log {} would drop anything but the aggregated result
10) the destination only receives the output of the entire log {} statement, e.g. the result of the 3rd aggregation
now as I think of it, the sample can even be improved so that the MID based grouping-by (e.g. the 2nd aggregation) does not receive the output of the first one. The two aggregation both get a copy of the incoming data, and their combined output is fed through the 3rd aggregation
but I leave you to it, this is just improvement in performance, functionally it should work just like that.
I am having problems with getting shared libraries to load. I am running rsyslog (corporate standard with a config I can't change) and syslog-ng (for my stuff) on an ubuntu 20.04 system
I have installed syslog-ng from the "unofficial" packages in /usr/local and fiddled with the systemd config to get the libraries loaded and it was all working.
now something has changed and I get the error:
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: Error parsing source statement, source plugin network not found in /etc/syslog-ng/conf.d/eset.conf:2:9-2:16:
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 1 source s_eset {
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 2-----> network(transport("tcp") port(5514) keep-alive(yes) max_connections(2));
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 2-----> ^^^^^^^
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 3 };I assume the issue is that syslog-ng is not finding the library with the source network plugin.
I have these vars set in /etc/default/syslog-ng
SYSLOGNG_OPTS="--control /var/lib/syslog-sec/syslog-ng.ctl --module-path /usr/local/lib/syslog-ng/3.31 --persist-file /var/lib/syslog-sec/syslog-ng.persist --pidfile /var/lib/syslog-sec/syslog-ng.pid"
LD_LIBRARY_PATH="/usr/local/lib/syslog-ng"Any thoughts on what might be wrong?
Ah! this may be it:
Error opening plugin module; module='afsocket', error='libnet.so.1: cannot open shared object file: No such file or directory'
The affile module provides file source & destination support for syslog-ng.
THe system was rebooted and immediately after that it started to complain that it could not find libivykis.so.0. I fixed that about a week ago (can't remember how and
history is not helpful). Then it started loading the config and barfing on the network plugin. THat was a week ago and I have just got back to it.
I think what happened was that in my attempts to get syslog-ng and rsyslog to coexist on an ubuntu system I had installed and uninstall both at least twice. I ended up installing the package from open SUSE and moving stuff around in the install directories to stop ubuntu deleting them.
I ended up this morning of going back to scratch and building a package from source that installed in /usr/local/ bin (this time with systemd support which is what my original one lacked). That runs fine so long as I include /usr/local/lib in LD_LIBRARY_PATH
any. ideas on debugging this?
Can you show the error message? I havent seen anything like this, unless its some kind of dependency issue. Btw have you tried to compile it using our docker based build infrastructure? Or even getting the binary from github? The packaging files can be customized if the ones we have dont match your use case. Or if you want I can help in just producing the binaries in a tarball.
here is the log from syslog-ng:
Jan 22 08:29:04 secmgrprd01 syslog-ng[2218448]: syslog-ng starting up; version='3.35.1'
Jan 22 08:30:34 secmgrprd01 syslog-ng[2218448]: syslog-ng shutting down; version='3.35.1'
Jan 22 08:30:34 secmgrprd01 syslog-ng[2218448]: Child program exited, restarting; cmdline='/usr/local/tools/dev/siem_logging/bin/syslog-ng-es.rb -vv --debug source --user sensors --source loghost', status='15'
hello ! .I am ashish tiwari , a computer science undergrad. I have just entered my second year of my college . I am new to open source contributions but I am well aware of C , C++ , Python , Javascript , Html , CSS and SQL.
can any one guide me how to get started?
Regards
Ashish
can any one guide me how to get started?
Regards
Ashish
once you have the source, just try
./dbld/rules pull-image-devshell, this should download the "devshell" image from the docker registry. If you use image-devshell argument it would build it locally.
Once you have the devshell image, you can enter by using
./dbld/rules shell, this will land you in an environment where all the build tools required for syslog-ng are already installed.
To compile there, go to the
/build directory and run /dbld/bootstrap
Once that finishes you can build using
make, you can install binaries within the docker container using make install
You can find the built binaries in the
/install directory within the container.
To start the syslog-ng you just built, try
/install/sbin/syslog-ng -Fe this should launch syslog-ng with the default configuration (which is installed into /install/etc)
I finally figured out the problem after leaving it to sit for a day. The fundamental issues turned out to be that the reboot had also removed some firewall rules so syslog-ng was not seeing the expected traffic. Doh! There are still some issues I am working through but things are generally working. BTW I found that if you are running rsyslog on ubuntu then you can not use systemd to manage the service even if you install syslog-ng without using
apt. The nasty bit is that systemctl start syslog-ng fails with a totally obscure error message Failed with result 'protocol'.. I eventually decided that this was ubuntu refusing to start syslog-ng because rsyslog was installed.
[announcement] I am doing a relaunch of the syslog-ng project. Read more on my blog about my plans, motivations & background. https://syslog-ng-future.blog/syslog-ng-relaunch/
That is not necessarily the only reason, the list server was hosted by balabit, and its existence was probably forgotten by One Identity, the company which acquired Balabit. I don't think it is very well maintained. Your email provider may simply not accepting messages from that server. I don't think DKIM or SPF would be configured.
The alternative might be to ask questions here or maybe github.
That's can be a reason. I have a very exotic email provider, Google :) So probably blocking dkim, spf is the reason... I hope sometimes we will have a working mailing list, but I'll ask questions here in the mean time. I'm also very excited on syslog-ng-future.blog, and hoping in a bright future of OSE :)
Let me ask something else and maybe ancient. What is the story behind rsyslog is in all the major distributions and not syslog-ng? I'm so tired to asking customers to remove rsyslog :) and I fully resist to learn rsyslog's weird config syntax :)
@vladx71:matrix.org The story was relatively simple, syslog-ng at 2009-10 was planned to go into Fedora as a default. However this was roughly the time when Balabit co-opted syslog-ng and started offering Premium Edition. At this time, "open core" and commercial addon to an open source project was a big no-no. Therefore Fedora8 cancelled their plan to go with syslog-ng and chose rsyslog instead.
rsyslog was a fork of sysklogd if I remember correctly, Rainer (from Adiscon) was also part of the IETF effort to create RFC5424. rsyslog was in a way another implementation of RFC5424 that was needed for a standard to be adopted.
One more thing I should ask on the disfunct mailing list :) One of my customers decided to use debian as the OS for log relay. It was ok for years, but now after a few hours syslog-ng start using 100% CPU. Config is not changed, we've tried to upgrade from Deb10 to 11, and also the latest OSE installed but it is still the same... where should I start looking for the reason of this?
This is a relatively simple setup with lots of sources but only two destinations (Qradar + Graylog) with standard network driver. EPS is not very high, it is about 1000 EPS
I have 10+ very similar server with similar configs, the only difference I can see, this is Debian
also when it starts killing the CPU, it stops processing logs as far as I see
2 replies
I'm fighting with some crazy vmware logsource and my filter is quite good, filtering out logs cannot be processed by my SIEM based on the program field (in-list filter). Unfortunately there are lots of messages dumped by this sytem with no real syslog structure (like tomcat java dumps). I wonder if there is any way to filter messages where for example program field not exists
here is an example message
<14>Feb 16 12:11:19 10.254.255.230 at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.processApiRequest(JsonServerConnection.java:396)
Can you identify the logsource somehow, like based on IP address? Because if you could you may be able to "fix" this message, like adding a program field, before sending it onwards. That's what I see in sc4s doing, fix the log so at the very least it goes to the right bucket/index/file. This message seems to be part of a java stacktrace.
yes, logsources identified by SRCIP. The problem is some logs are fine (standard *nix like logs) with perfect structure, but some part of the systems just drops texts in syslog without any header, so I'm thinking some filter where I can identify these messages like "program field is empty"