Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 00:39
    lucacome commented #4147
  • 00:38
    lucacome commented #3987
  • 00:30
    MrAnno commented #4147
  • 00:23
    MrAnno commented #4147
  • Sep 26 23:30
    lucacome commented #4147
  • Sep 26 17:56
    ryanfaircloth commented #4133
  • Sep 26 16:59
    MrAnno commented #4153
  • Sep 26 15:12
    ryanfaircloth opened #4153
  • Sep 26 15:12
    ryanfaircloth labeled #4153
  • Sep 26 11:30
    MrAnno closed #4149
  • Sep 26 11:30
    MrAnno locked #4149
  • Sep 26 10:30
    gaborznagy closed #4149
  • Sep 26 10:30
    gaborznagy commented #4149
  • Sep 26 07:59
    ramtech123 commented #4082
  • Sep 26 05:23
    daemondong commented #4149
  • Sep 25 19:56

    MrAnno on master

    syslog-ng-debun: Drop direct ac… Merge pull request #4144 from m… (compare)

  • Sep 25 19:56
    MrAnno closed #4139
  • Sep 25 19:56
    MrAnno closed #4144
  • Sep 25 19:30
    MrAnno commented #4144
  • Sep 25 19:20
    mochrul commented #4144
Balazs Scheidler
@bazsi
I hope it is clear up to this point.
With that all said, your synthetic message at the top of the context contains the ${attachment} field, inherited from any original message that we have seen so far, but it's going to contain the last of these values.
This means that even though you had only 2 attachments, "try.odt" and "house.html", your synthetic message would also contain $attachment and its value would be house.html, since that's the last message in the context.
$(context-lookup) iterates both the original messages and the synthetic one added on top, meaning house.html would be repeated.
Balazs Scheidler
@bazsi
I agree this is more complicated than it should be.
The solution to your immediate problem is to cut off the final element from the list returned by $(context-lookup), e.g. something like $(list-slice 0:-1 $(context-lookup ('$attachment' ne '') ${attachment}))
I'll check that expression in a minute.
Balazs Scheidler
@bazsi
The longer term solution is to make this a bit more intuitive. I was thinking on making $(context-lookup) ignore the last element of the context. The problem with this is that it is an incompatible change, one that is difficult to communicate and fix. The other problem with this approach is that only grouping-by() (and db-parser()) aggregation is what adds this extra, synthetic message in the context and there are other cases which don't have a synthetic message generated (e.g. simple matching rules in db-parser()).
Balazs Scheidler
@bazsi
Another solution would be to have an argument to $(context-lookup) and $(context-values) to ignore the synthetic message, which would at least make the expression a bit easier to come up with and read.
Balazs Scheidler
@bazsi
A 3rd solution is to simply document this better.
arteta22000
@arteta22000
Thanks @bazsi for the explanation.
I have another question. With these ugly cisco ironport logs. There are 3 different ids, MID is the main id. In the raw log example above, we see that a session can start with "ICID" ( New SMTP ICID 219986669 ...) and then another log does the mapping (ICID <> MID , Start MID 198090335 . ICID 219986669 ...).
Can I aggregate the logs with ICID and DCID and then aggregate with MID. I don't know if I'm clear?
arteta22000
@arteta22000
I repost an example of a log session:

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: New SMTP ICID 219986669 interface responsibility (8.16.8.16) address 17.39.24.2 reverse dns host cuis.com verified no

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: ICID 219986669 RELAY SG UNKNOWNLIST match sbrs[0.0:7.0] SBRS 5.1

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Start MID 198090335 ICID 219986669

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ICID 219986669 RID 0 To: fcvez@yaho.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ready 12710 bytes from tgamb@hotil.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ICID 219986669 RID 0 To: fcvez@yaho.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 Subject 'Which add raise.'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 Message-ID 'tgamb@hotil.com'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 antivirus negative

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 attachment 'try.odt'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 attachment 'house.html'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Delivery start DCID 177209764 MID 198090335 to RID [0]

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 RID [0] Response '2.6.0 22633284F46CFCC33@mray.com [InternalId=4230] Queued mail for delivery'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 using engine: CASE spam positive

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 rewritten to MID 204028130 by add-footer filter 'Footer Stamping'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Message done DCID 158099468 MID 198090335 to RID [0] ['([168.xxxx.137])\r\n by hester.org with ESMTP; 11 Aug 1976 17:03:41 +0200)']

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Message finished MID 198090335 done

Balazs Scheidler
@bazsi
This is possible to do in db-parser() but that's pretty arcane to use with its XML based format. @faxm0dem created a puppet wrapper for it and yaml based generation that is a lot easier to use.
The keyword to connect two independent correlation state is called the <create-context> action. Here's a bit of documentation for that:
Example: Creating a new context from an action

In syslog-ng OSE version 3.8 and newer, you can create a new context as an action. For details, see Element: create-context.

The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the <create-context> element.

<rule provider='test' id='12' class='violation'>
  <patterns>
    <pattern>simple-message-with-action-to-create-context</pattern>
  </patterns>
  <actions>
    <action trigger='match'>
      <create-context context-id='1000' context-timeout='60' context-scope='program'>
        <message inherit-properties='context'>
          <values>
            <value name='MESSAGE'>context message</value>
          </values>
        </message>
      </create-context>
    </action>
  </actions>
</rule>
In the case of db-parser() you are not limited to simply stuffing messages into a "context" that grouping-by() does, rather you have individual rules that can act based on the input.
This is a powerful feature but one that is pretty difficult to use. It honours your efforts with performance though.
What you would need to do basically is the following:
1) match the beginning of the session and correlate on ${CID}
Balazs Scheidler
@bazsi
2) match the "Start CID ... MID ..." message and on that create a context, this will create a NEW correlation context with a key that is present in the current message (e.g. ${MID}), in this case you can generate a synthetic message that becomes part of your new correlation state.
3) as you are matching the rest of the session (e.g. which contain ${MID}), you correlate against ${MID} and you will find a very-first synthetic message in this context. The one that you created as a result of the first point, e.g. the CID based correlation. Here you would need to represent all CID based data as a single message. With syslog-ng lists, you could encapsulate all CID based messages into a single syslog-ng list. e.g $(context-values $MSG) as a name-value pair. but if you have name-value pairs to extract, that might be a better solution.
hope this helps.
arteta22000
@arteta22000
Thank you, it seems complex for me. I preferred the solution of aggregating cid/mid/dcid and then aggregating the whole.
Currently; we do this with a python script and redis, but we have performance problems.
But I can't do it with grouping-by ? with several grouping-by subsequent
Balazs Scheidler
@bazsi
you are right. that could work.
I didn't think of that.
arteta22000
@arteta22000
How do I do this? I can't fowardthe aggregation from one grouping-by to another.
Balazs Scheidler
@bazsi
if you performed grouping-by()s separately: 1) one context CID stuff (and the Start CID/MID message), 2) one for the MID based messages plus the Start CID/MID message), then both contexts would have both MID/CID information. Then a 2nd layer of grouping-by() could correlate the aggregation of the 1st and the 2nd.
exactly.
good idea on using two. I was just too focused on doing it in one step.
arteta22000
@arteta22000
But how to send the aggregation from one grouping-by to another? I have the feeling that "grouping-by" works by default with the "message" macro?
example:

parser groupingby {
grouping-by(
key("${cid_id}")
scope("HOST")
value("event.aggregate" "ok")
value("MESSAGE" "Session completed >> ${attachment} cid_id:${cid_id} info1: ${info1}}")
inherit-mode("context")
)
timeout(10)
inject-mode("pass-through")
);
};

parser groupingby2 {
grouping-by(
key("${mid_id}")
scope("HOST")
trigger( "${.classifier.rule_id}" eq "regex_finished_status" )
having( "${finished_status}" eq "done" )
aggregate(
value("event.aggregate" "ok")
value("MESSAGE" "Session completed >> ${attachment} mid_id:${mid_id} cip_icid: ${cip_icid} ironport_interface_name: ${ironport_interface_name} cid_id:${cid_id} info1: ${info1}}")
inherit-mode("context")
)
timeout(10)
inject-mode("pass-through")
);
};

arteta22000
@arteta22000
With "inject-mode("internal")" in the first grouping-by ?
Balazs Scheidler
@bazsi
inject-mode(internal) would generate the message as if it was coming from the internal() source
I think inject-mode(passthrough) is better, that way the message is generated as if coming from grouping-by() itself.
arteta22000
@arteta22000

I'm going to do some tests but I'm a bit lost,)

My first "grouping-by" will generate a macro (that I configure; in the value parameter) with the aggregation (example: mid :xxx cid:xxx source_smtp: xxxx).

How to forward this information (cid:xxx source_smtp: xxxx) to the other grouping-by (after) so that it aggregates this information with the central mid
Balazs Scheidler
@bazsi
just connect the two grouping-by()s in a log path, e.g.
log { source(whatever); parser { grouping-by(FIRST...); grouping-by(SECOND...); }; ... };
you can also define it as a top-level parser block and then reference it:
parser p_ironport {
    grouping-by(FIRST...);
   grouping-by(SECOND...);
};

log { source(whatever); parser(p_ironport); ... };
any element can be concatenated to another one, each would be processed in order.
NOTE: in the first example I used braces to indicate that the parsers are defined in-line, within the log statement. In the 2nd example, I used parenthesis which references a parser that was declared earlier.
so parser { in-line-parser-expression }; or parser(namedparserblock);
arteta22000
@arteta22000
I tried, and I got 2 aggregated logs in output. I need to put "value("MESSAGE")" in the first grouping-by block?
The second grouping-by will see the message aggregated using the message macro ?
Balazs Scheidler
@bazsi
I don't really understand the question. I'd try something like this (not syntax checked, just from the top-of-my-head):

log {
    source(whatever);
   parser { db-parser(file('ironport.xml')); };
    log {
        filter { <match only iron port logs...>; };

        if ('${CID}' ne '') {
            parser { grouping-by(<CID aggregation options> inject-mode(passthrough)); };
        };
        if ('${MID}' ne '') {
            parser { grouping-by(<MID aggregation options> inject-mode(passthrough)); };
        };
        if (<match aggregated message from either CID or MID based aggregation>) {
            parser { grouping-by(<MID + CID result aggregation> inject-mode(passthrough)); };
        };
       filter { <match only aggregated logs emitted by the last grouping-by()>); };
    };
    destination(whatever);
};