syslog-ng/syslog-ng

syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.

Balazs Scheidler

@bazsi

parser p_ironport {
    grouping-by(FIRST...);
   grouping-by(SECOND...);
};

log { source(whatever); parser(p_ironport); ... };

any element can be concatenated to another one, each would be processed in order.

NOTE: in the first example I used braces to indicate that the parsers are defined in-line, within the log statement. In the 2nd example, I used parenthesis which references a parser that was declared earlier.

so parser { in-line-parser-expression }; or parser(namedparserblock);

arteta22000

@arteta22000

I tried, and I got 2 aggregated logs in output. I need to put "value("MESSAGE")" in the first grouping-by block?

The second grouping-by will see the message aggregated using the message macro ?

Balazs Scheidler

@bazsi

I don't really understand the question. I'd try something like this (not syntax checked, just from the top-of-my-head):


log {
    source(whatever);
   parser { db-parser(file('ironport.xml')); };
    log {
        filter { <match only iron port logs...>; };

        if ('${CID}' ne '') {
            parser { grouping-by(<CID aggregation options> inject-mode(passthrough)); };
        };
        if ('${MID}' ne '') {
            parser { grouping-by(<MID aggregation options> inject-mode(passthrough)); };
        };
        if (<match aggregated message from either CID or MID based aggregation>) {
            parser { grouping-by(<MID + CID result aggregation> inject-mode(passthrough)); };
        };
       filter { <match only aggregated logs emitted by the last grouping-by()>); };
    };
    destination(whatever);
};

Balazs Scheidler

@bazsi

the messages flow through the sequence one by one.
1) The source() statement in the front pulls in any messages received by the specific source.
2) the db-parser() right next would extract ironport name-value pairs from the messages
3) the embedded log {} statement just encapsulates a set of operations
4) the first filter embedded in the log {} statement would drop anything but ironport (I am not sore if the db-parser() field extraction sets a specific field or not, but that's certainly doable based on some criteria)
5) the first if() checks if $CID is set, if it is, a grouping-by() parser is executed, which should in turn emit an aggregated result of all messages where CID was set.
6) the second if() checks if $MID is set, similarly to the first, it would need to aggregate the $MID related messages into a result
7) the last if() would match messages that either the first or the second grouping-by() emitted and runs a 3rd grouping-by()
8) this third grouping-by() is our end result
9) the filter as the last statement within our log {} would drop anything but the aggregated result
10) the destination only receives the output of the entire log {} statement, e.g. the result of the 3rd aggregation

Balazs Scheidler

@bazsi

now as I think of it, the sample can even be improved so that the MID based grouping-by (e.g. the 2nd aggregation) does not receive the output of the first one. The two aggregation both get a copy of the incoming data, and their combined output is fed through the 3rd aggregation

but I leave you to it, this is just improvement in performance, functionally it should work just like that.

Russell Fulton

@rful011

I am having problems with getting shared libraries to load. I am running rsyslog (corporate standard with a config I can't change) and syslog-ng (for my stuff) on an ubuntu 20.04 system
I have installed syslog-ng from the "unofficial" packages in /usr/local and fiddled with the systemd config to get the libraries loaded and it was all working.

now something has changed and I get the error:

Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: Error parsing source statement, source plugin network not found in /etc/syslog-ng/conf.d/eset.conf:2:9-2:16:
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 1       source s_eset {
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 2----->         network(transport("tcp") port(5514) keep-alive(yes) max_connections(2));
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 2----->         ^^^^^^^
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 3           };

I assume the issue is that syslog-ng is not finding the library with the source network plugin.
I have these vars set in /etc/default/syslog-ng

SYSLOGNG_OPTS="--control /var/lib/syslog-sec/syslog-ng.ctl --module-path /usr/local/lib/syslog-ng/3.31 --persist-file /var/lib/syslog-sec/syslog-ng.persist --pidfile /var/lib/syslog-sec/syslog-ng.pid"
LD_LIBRARY_PATH="/usr/local/lib/syslog-ng"

Any thoughts on what might be wrong?

Balazs Scheidler

@bazsi

Does it run in an interactive shell? What does syslog-ng --module-registry tell you with that module-path argument?

Russell Fulton

@rful011

yes, done that and it produces lots of stuff -- what should I look for?

Ah! this may be it:

Error opening plugin module; module='afsocket', error='libnet.so.1: cannot open shared object file: No such file or directory'
  The affile module provides file source & destination support for syslog-ng.

Russell Fulton

@rful011

BTW I have figured out what "changed".

Russell Fulton

@rful011

THe system was rebooted and immediately after that it started to complain that it could not find libivykis.so.0. I fixed that about a week ago (can't remember how and history is not helpful). Then it started loading the config and barfing on the network plugin. THat was a week ago and I have just got back to it.

Balazs Scheidler

@bazsi

libnet is needed for spoof-source support, it is not part of ivykis

Russell Fulton

@rful011

solved this one -- libnet was missing The tricky bit was figuring out that the package is called libnet1.

I think what happened was that in my attempts to get syslog-ng and rsyslog to coexist on an ubuntu system I had installed and uninstall both at least twice. I ended up installing the package from open SUSE and moving stuff around in the install directories to stop ubuntu deleting them.

Russell Fulton

@rful011

I got it working but then out "auto patcher" ran and the system rebooted. I suspect something in that process spotted that libnet had been installed as a dependency of a package that was no longer installed and removed it

I ended up this morning of going back to scratch and building a package from source that installed in /usr/local/ bin (this time with systemd support which is what my original one lacked). That runs fine so long as I include /usr/local/lib in LD_LIBRARY_PATH

Russell Fulton

@rful011

Oh dear. My locally compiled version starts fine but fails to pass any data to a program destination. The program starts and reads STDIN but never gets anything. The syslog daemon eventually terminates with a Timeout error -- presumably on the write to the pipe?

any. ideas on debugging this?

Balazs Scheidler

@bazsi

Can you show the error message? I havent seen anything like this, unless its some kind of dependency issue. Btw have you tried to compile it using our docker based build infrastructure? Or even getting the binary from github? The packaging files can be customized if the ones we have dont match your use case. Or if you want I can help in just producing the binaries in a tarball.

Russell Fulton

@rful011

Thanks Balazs! I don't have any experience with docker :(

here is the log from syslog-ng:

Jan 22 08:29:04 secmgrprd01 syslog-ng[2218448]: syslog-ng starting up; version='3.35.1'
Jan 22 08:30:34 secmgrprd01 syslog-ng[2218448]: syslog-ng shutting down; version='3.35.1'
Jan 22 08:30:34 secmgrprd01 syslog-ng[2218448]: Child program exited, restarting; cmdline='/usr/local/tools/dev/siem_logging/bin/syslog-ng-es.rb -vv --debug source --user sensors --source loghost', status='15'

Russell Fulton

@rful011

I will come back to this with more info -- got to run right
now

Ashish Tiwari

@Ashish-100-tiwari

hello ! .I am ashish tiwari , a computer science undergrad. I have just entered my second year of my college . I am new to open source contributions but I am well aware of C , C++ , Python , Javascript , Html , CSS and SQL.
can any one guide me how to get started?
Regards
Ashish

Balazs Scheidler

@bazsi

@Ashish-100-tiwari try cloning the source code from github and prepare a work environment by building the code yourself. The easiest way is probably using dbld, a docker based build system.

once you have the source, just try ./dbld/rules pull-image-devshell, this should download the "devshell" image from the docker registry. If you use image-devshell argument it would build it locally.

Once you have the devshell image, you can enter by using ./dbld/rules shell, this will land you in an environment where all the build tools required for syslog-ng are already installed.

To compile there, go to the /build directory and run /dbld/bootstrap

Once that finishes you can build using make, you can install binaries within the docker container using make install

You can find the built binaries in the /install directory within the container.

To start the syslog-ng you just built, try /install/sbin/syslog-ng -Fe this should launch syslog-ng with the default configuration (which is installed into /install/etc)

Russell Fulton

@rful011

I finally figured out the problem after leaving it to sit for a day. The fundamental issues turned out to be that the reboot had also removed some firewall rules so syslog-ng was not seeing the expected traffic. Doh! There are still some issues I am working through but things are generally working. BTW I found that if you are running rsyslog on ubuntu then you can not use systemd to manage the service even if you install syslog-ng without using apt. The nasty bit is that systemctl start syslog-ng fails with a totally obscure error message Failed with result 'protocol'.. I eventually decided that this was ubuntu refusing to start syslog-ng because rsyslog was installed.

Balazs Scheidler

@bazsi

[announcement] I am doing a relaunch of the syslog-ng project. Read more on my blog about my plans, motivations & background. https://syslog-ng-future.blog/syslog-ng-relaunch/

Laszlo Pal

@Vladx71

Hi guys, I'm trying to subscribe to the OSE mailing list, but it seems I'm banned because I never receive the confirmation mail. I was a member for decades, but it seems I've lost my membership

Balazs Scheidler

@bazsi

That is not necessarily the only reason, the list server was hosted by balabit, and its existence was probably forgotten by One Identity, the company which acquired Balabit. I don't think it is very well maintained. Your email provider may simply not accepting messages from that server. I don't think DKIM or SPF would be configured.

The alternative might be to ask questions here or maybe github.

Laszlo Pal

@vladx71:matrix.org

[m]

That's can be a reason. I have a very exotic email provider, Google :) So probably blocking dkim, spf is the reason... I hope sometimes we will have a working mailing list, but I'll ask questions here in the mean time. I'm also very excited on syslog-ng-future.blog, and hoping in a bright future of OSE :)

Let me ask something else and maybe ancient. What is the story behind rsyslog is in all the major distributions and not syslog-ng? I'm so tired to asking customers to remove rsyslog :) and I fully resist to learn rsyslog's weird config syntax :)

Gábor Nagy

@gaborznagy

Hi @Vladx71 We haven't forgotten the OSE mailing list. I would like to confirm that have experienced issues with gmail addresses.

Balazs Scheidler

@bazsi

@vladx71:matrix.org The story was relatively simple, syslog-ng at 2009-10 was planned to go into Fedora as a default. However this was roughly the time when Balabit co-opted syslog-ng and started offering Premium Edition. At this time, "open core" and commercial addon to an open source project was a big no-no. Therefore Fedora8 cancelled their plan to go with syslog-ng and chose rsyslog instead.

rsyslog was a fork of sysklogd if I remember correctly, Rainer (from Adiscon) was also part of the IETF effort to create RFC5424. rsyslog was in a way another implementation of RFC5424 that was needed for a standard to be adopted.

Laszlo Pal

@vladx71:matrix.org

[m]

One more thing I should ask on the disfunct mailing list :) One of my customers decided to use debian as the OS for log relay. It was ok for years, but now after a few hours syslog-ng start using 100% CPU. Config is not changed, we've tried to upgrade from Deb10 to 11, and also the latest OSE installed but it is still the same... where should I start looking for the reason of this?

This is a relatively simple setup with lots of sources but only two destinations (Qradar + Graylog) with standard network driver. EPS is not very high, it is about 1000 EPS

I have 10+ very similar server with similar configs, the only difference I can see, this is Debian

Laszlo Pal

@vladx71:matrix.org

[m]

also when it starts killing the CPU, it stops processing logs as far as I see

Where communities thrive

People

Repo info

Activity