Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 17:06
    MrAnno commented #4235
  • 17:05
    MrAnno labeled #4235
  • 17:05
    MrAnno unlabeled #4235
  • 15:29
    shamine starred syslog-ng/syslog-ng
  • 15:03
    kira-syslogng commented #4204
  • 14:37
    HofiOne commented #4204
  • 12:35
    kira-syslogng commented #4204
  • 12:08
    HofiOne commented #4204
  • 11:12
    kira-syslogng commented #4204
  • 10:46
    HofiOne commented #4204
  • 10:45
    kira-syslogng commented #4204
  • 10:19
    HofiOne commented #4204
  • 09:35
    nkl0x55 starred syslog-ng/syslog-ng
  • 09:34
    kira-syslogng commented #4204
  • 09:08
    HofiOne commented #4204
  • 07:39
    duminghui1985 labeled #4235
  • 07:39
    duminghui1985 opened #4235
  • Dec 05 17:53
    github-actions[bot] commented #4204
  • Dec 05 17:53

    MrAnno on master

    csv-scanner: add support for CS… csv-scanner: added more unit te… csv-parser: fix parsing of the … and 7 more (compare)

  • Dec 05 17:53
    MrAnno closed #4228
Balazs Scheidler
@bazsi
Another solution would be to have an argument to $(context-lookup) and $(context-values) to ignore the synthetic message, which would at least make the expression a bit easier to come up with and read.
Balazs Scheidler
@bazsi
A 3rd solution is to simply document this better.
arteta22000
@arteta22000
Thanks @bazsi for the explanation.
I have another question. With these ugly cisco ironport logs. There are 3 different ids, MID is the main id. In the raw log example above, we see that a session can start with "ICID" ( New SMTP ICID 219986669 ...) and then another log does the mapping (ICID <> MID , Start MID 198090335 . ICID 219986669 ...).
Can I aggregate the logs with ICID and DCID and then aggregate with MID. I don't know if I'm clear?
arteta22000
@arteta22000
I repost an example of a log session:

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: New SMTP ICID 219986669 interface responsibility (8.16.8.16) address 17.39.24.2 reverse dns host cuis.com verified no

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: ICID 219986669 RELAY SG UNKNOWNLIST match sbrs[0.0:7.0] SBRS 5.1

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Start MID 198090335 ICID 219986669

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ICID 219986669 RID 0 To: fcvez@yaho.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ready 12710 bytes from tgamb@hotil.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ICID 219986669 RID 0 To: fcvez@yaho.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 Subject 'Which add raise.'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 Message-ID 'tgamb@hotil.com'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 antivirus negative

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 attachment 'try.odt'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 attachment 'house.html'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Delivery start DCID 177209764 MID 198090335 to RID [0]

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 RID [0] Response '2.6.0 22633284F46CFCC33@mray.com [InternalId=4230] Queued mail for delivery'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 using engine: CASE spam positive

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 rewritten to MID 204028130 by add-footer filter 'Footer Stamping'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Message done DCID 158099468 MID 198090335 to RID [0] ['([168.xxxx.137])\r\n by hester.org with ESMTP; 11 Aug 1976 17:03:41 +0200)']

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Message finished MID 198090335 done

Balazs Scheidler
@bazsi
This is possible to do in db-parser() but that's pretty arcane to use with its XML based format. @faxm0dem created a puppet wrapper for it and yaml based generation that is a lot easier to use.
The keyword to connect two independent correlation state is called the <create-context> action. Here's a bit of documentation for that:
Example: Creating a new context from an action

In syslog-ng OSE version 3.8 and newer, you can create a new context as an action. For details, see Element: create-context.

The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the <create-context> element.

<rule provider='test' id='12' class='violation'>
  <patterns>
    <pattern>simple-message-with-action-to-create-context</pattern>
  </patterns>
  <actions>
    <action trigger='match'>
      <create-context context-id='1000' context-timeout='60' context-scope='program'>
        <message inherit-properties='context'>
          <values>
            <value name='MESSAGE'>context message</value>
          </values>
        </message>
      </create-context>
    </action>
  </actions>
</rule>
In the case of db-parser() you are not limited to simply stuffing messages into a "context" that grouping-by() does, rather you have individual rules that can act based on the input.
This is a powerful feature but one that is pretty difficult to use. It honours your efforts with performance though.
What you would need to do basically is the following:
1) match the beginning of the session and correlate on ${CID}
Balazs Scheidler
@bazsi
2) match the "Start CID ... MID ..." message and on that create a context, this will create a NEW correlation context with a key that is present in the current message (e.g. ${MID}), in this case you can generate a synthetic message that becomes part of your new correlation state.
3) as you are matching the rest of the session (e.g. which contain ${MID}), you correlate against ${MID} and you will find a very-first synthetic message in this context. The one that you created as a result of the first point, e.g. the CID based correlation. Here you would need to represent all CID based data as a single message. With syslog-ng lists, you could encapsulate all CID based messages into a single syslog-ng list. e.g $(context-values $MSG) as a name-value pair. but if you have name-value pairs to extract, that might be a better solution.
hope this helps.
arteta22000
@arteta22000
Thank you, it seems complex for me. I preferred the solution of aggregating cid/mid/dcid and then aggregating the whole.
Currently; we do this with a python script and redis, but we have performance problems.
But I can't do it with grouping-by ? with several grouping-by subsequent
Balazs Scheidler
@bazsi
you are right. that could work.
I didn't think of that.
arteta22000
@arteta22000
How do I do this? I can't fowardthe aggregation from one grouping-by to another.
Balazs Scheidler
@bazsi
if you performed grouping-by()s separately: 1) one context CID stuff (and the Start CID/MID message), 2) one for the MID based messages plus the Start CID/MID message), then both contexts would have both MID/CID information. Then a 2nd layer of grouping-by() could correlate the aggregation of the 1st and the 2nd.
exactly.
good idea on using two. I was just too focused on doing it in one step.
arteta22000
@arteta22000
But how to send the aggregation from one grouping-by to another? I have the feeling that "grouping-by" works by default with the "message" macro?
example:

parser groupingby {
grouping-by(
key("${cid_id}")
scope("HOST")
value("event.aggregate" "ok")
value("MESSAGE" "Session completed >> ${attachment} cid_id:${cid_id} info1: ${info1}}")
inherit-mode("context")
)
timeout(10)
inject-mode("pass-through")
);
};

parser groupingby2 {
grouping-by(
key("${mid_id}")
scope("HOST")
trigger( "${.classifier.rule_id}" eq "regex_finished_status" )
having( "${finished_status}" eq "done" )
aggregate(
value("event.aggregate" "ok")
value("MESSAGE" "Session completed >> ${attachment} mid_id:${mid_id} cip_icid: ${cip_icid} ironport_interface_name: ${ironport_interface_name} cid_id:${cid_id} info1: ${info1}}")
inherit-mode("context")
)
timeout(10)
inject-mode("pass-through")
);
};

arteta22000
@arteta22000
With "inject-mode("internal")" in the first grouping-by ?
Balazs Scheidler
@bazsi
inject-mode(internal) would generate the message as if it was coming from the internal() source
I think inject-mode(passthrough) is better, that way the message is generated as if coming from grouping-by() itself.
arteta22000
@arteta22000

I'm going to do some tests but I'm a bit lost,)

My first "grouping-by" will generate a macro (that I configure; in the value parameter) with the aggregation (example: mid :xxx cid:xxx source_smtp: xxxx).

How to forward this information (cid:xxx source_smtp: xxxx) to the other grouping-by (after) so that it aggregates this information with the central mid
Balazs Scheidler
@bazsi
just connect the two grouping-by()s in a log path, e.g.
log { source(whatever); parser { grouping-by(FIRST...); grouping-by(SECOND...); }; ... };
you can also define it as a top-level parser block and then reference it:
parser p_ironport {
    grouping-by(FIRST...);
   grouping-by(SECOND...);
};

log { source(whatever); parser(p_ironport); ... };
any element can be concatenated to another one, each would be processed in order.
NOTE: in the first example I used braces to indicate that the parsers are defined in-line, within the log statement. In the 2nd example, I used parenthesis which references a parser that was declared earlier.
so parser { in-line-parser-expression }; or parser(namedparserblock);
arteta22000
@arteta22000
I tried, and I got 2 aggregated logs in output. I need to put "value("MESSAGE")" in the first grouping-by block?
The second grouping-by will see the message aggregated using the message macro ?
Balazs Scheidler
@bazsi
I don't really understand the question. I'd try something like this (not syntax checked, just from the top-of-my-head):

log {
    source(whatever);
   parser { db-parser(file('ironport.xml')); };
    log {
        filter { <match only iron port logs...>; };

        if ('${CID}' ne '') {
            parser { grouping-by(<CID aggregation options> inject-mode(passthrough)); };
        };
        if ('${MID}' ne '') {
            parser { grouping-by(<MID aggregation options> inject-mode(passthrough)); };
        };
        if (<match aggregated message from either CID or MID based aggregation>) {
            parser { grouping-by(<MID + CID result aggregation> inject-mode(passthrough)); };
        };
       filter { <match only aggregated logs emitted by the last grouping-by()>); };
    };
    destination(whatever);
};
Balazs Scheidler
@bazsi
the messages flow through the sequence one by one.
1) The source() statement in the front pulls in any messages received by the specific source.
2) the db-parser() right next would extract ironport name-value pairs from the messages
3) the embedded log {} statement just encapsulates a set of operations
4) the first filter embedded in the log {} statement would drop anything but ironport (I am not sore if the db-parser() field extraction sets a specific field or not, but that's certainly doable based on some criteria)
5) the first if() checks if $CID is set, if it is, a grouping-by() parser is executed, which should in turn emit an aggregated result of all messages where CID was set.
6) the second if() checks if $MID is set, similarly to the first, it would need to aggregate the $MID related messages into a result
7) the last if() would match messages that either the first or the second grouping-by() emitted and runs a 3rd grouping-by()
8) this third grouping-by() is our end result
9) the filter as the last statement within our log {} would drop anything but the aggregated result
10) the destination only receives the output of the entire log {} statement, e.g. the result of the 3rd aggregation
Balazs Scheidler
@bazsi
now as I think of it, the sample can even be improved so that the MID based grouping-by (e.g. the 2nd aggregation) does not receive the output of the first one. The two aggregation both get a copy of the incoming data, and their combined output is fed through the 3rd aggregation
but I leave you to it, this is just improvement in performance, functionally it should work just like that.
Russell Fulton
@rful011

I am having problems with getting shared libraries to load. I am running rsyslog (corporate standard with a config I can't change) and syslog-ng (for my stuff) on an ubuntu 20.04 system
I have installed syslog-ng from the "unofficial" packages in /usr/local and fiddled with the systemd config to get the libraries loaded and it was all working.

now something has changed and I get the error:

Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: Error parsing source statement, source plugin network not found in /etc/syslog-ng/conf.d/eset.conf:2:9-2:16:
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 1       source s_eset {
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 2----->         network(transport("tcp") port(5514) keep-alive(yes) max_connections(2));
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 2----->         ^^^^^^^
Jan 18 08:01:45 secmgrprd01 syslog-ng[1831033]: 3           };

I assume the issue is that syslog-ng is not finding the library with the source network plugin.
I have these vars set in /etc/default/syslog-ng

SYSLOGNG_OPTS="--control /var/lib/syslog-sec/syslog-ng.ctl --module-path /usr/local/lib/syslog-ng/3.31 --persist-file /var/lib/syslog-sec/syslog-ng.persist --pidfile /var/lib/syslog-sec/syslog-ng.pid"
LD_LIBRARY_PATH="/usr/local/lib/syslog-ng"

Any thoughts on what might be wrong?

Balazs Scheidler
@bazsi
Does it run in an interactive shell? What does syslog-ng --module-registry tell you with that module-path argument?
Russell Fulton
@rful011
yes, done that and it produces lots of stuff -- what should I look for?
Ah! this may be it: Error opening plugin module; module='afsocket', error='libnet.so.1: cannot open shared object file: No such file or directory' The affile module provides file source & destination support for syslog-ng.
Russell Fulton
@rful011
BTW I have figured out what "changed".