Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 10:10
    MrAnno review_requested #4081
  • Aug 12 13:36
    Kokan closed #4100
  • Aug 12 13:36

    Kokan on master

    template: fix assertion failure… Merge pull request #4101 from b… (compare)

  • Aug 12 13:36
    Kokan closed #4101
  • Aug 12 11:04

    OverOrion on master

    secure-logging: fix possible in… Merge pull request #4102 from M… (compare)

  • Aug 12 11:04
    OverOrion closed #4102
  • Aug 12 10:37
    OverOrion review_requested #4102
  • Aug 12 10:35
    github-actions[bot] commented #4102
  • Aug 12 10:35
    MrAnno edited #4102
  • Aug 12 10:35
    MrAnno opened #4102
  • Aug 11 18:45
    ffontaine synchronize #4081
  • Aug 11 13:32
    ffontaine synchronize #4081
  • Aug 11 09:13
    Kokan commented #4101
  • Aug 10 15:22
    bazsi commented #4101
  • Aug 10 08:39
    Kokan commented #4101
  • Aug 09 17:41
    naltun starred syslog-ng/syslog-ng
  • Aug 09 14:51
    jorikseldeslachts starred syslog-ng/syslog-ng
  • Aug 09 08:35
    mohitvaid commented #4013
  • Aug 08 18:12
    kira-syslogng commented #4101
  • Aug 08 17:47
    github-actions[bot] commented #4101
deepulikesbru
@deepulikesbru
@MrAnno @gaborznagy
Above is the config file, When i try to give another file name say myfile3 after some testing, the deleted files also exist in the /var/log along with newly created log myfile3.
Gábor Nagy
@gaborznagy
@deepulikesbru Thanks for sharing the config!
The above config looks okay, but maybe your config is inavlid, as @MrAnno suggested it. Check it with a syslog-ng --syntax-only please.
What kind of syslog-ng do you use? Can you share us a syslog-ng -V output, please?
deepulikesbru
@deepulikesbru
Could @gaborznagy @MrAnno you please provide me the steps to reload the syslog configuration file properly? Thanks in advance
1 reply
deepulikesbru
@deepulikesbru
6.0.21 version AIX OS
The above command dont work in AIX
László Várady
@MrAnno

This channel is for the open-source edition of syslog-ng. Sorry, we can not support syslog-ng PE 6.0.21 here. AIX is a quite rare platform, please consider contacting the commercial support: https://support.oneidentity.com/syslog-ng-premium-edition/6.0.21

(/opt/syslog-ng/sbin/syslog-ng-ctl reload should probably work in this case)

deepulikesbru
@deepulikesbru
Thank you @MrAnno for your time on this. I will contact them.
Balazs Scheidler
@bazsi
@MrAnno while working on typing in PR #3831 I have a grammar conflict that I could resolve if plugin discovery would be required. Do you remember any case where plugin discovery was causing problems?
42 replies
arteta22000
@arteta22000
Hi, I'm testing the grouping-by feature for ugly logs: Cisco ironport (3 ID uniq: mid, dcid, lcid). Syslog-ng configuration here: https://www.codepile.net/pile/RLB9KA26
The "context-values" function returns duplicates, I don't understand ? there is only one log that matches in patterndb. Ouptut logs: Session completed >> "","","","","","",cold.wav,human.docx,do.mov,"","","","","","",do.mov mid_id:160681582 cip_icid: 298826272
Balazs Scheidler
@bazsi
Can you also publish sample messages?
arteta22000
@arteta22000
Sure. sample here: https://pastebin.com/r98CCzFa patterndb here: https://pastebin.com/Sw96MuRb . It's my first patterndb, I might have made mistakes
Balazs Scheidler
@bazsi
Thanks. I'll look into it
arteta22000
@arteta22000
Hi @bazsi , I've tried to reproduce my issue ?
Balazs Scheidler
@bazsi
15 of your 17 input lines do match the patterndb
Balazs Scheidler
@bazsi
So all the lines end up in your correlation context which have a ${mid_id} value
This means that the context contains the entire transaction and not just the lines that contain attachment information. For some of your log entries the name of the attachment is empty.
If all you want is to filter out messages that have ${attachment} defined, you could do this:
$(filter ('$_' ne '') $(context-values ${attachment}))
Or, maybe even simpler to use the $(context-lookup) function that combines $(filter) and $(context-values), like this
Balazs Scheidler
@bazsi
$(context-lookup ('$attachment' ne '') ${attachment})
Balazs Scheidler
@bazsi
@arteta22000 did ^^^ help?
arteta22000
@arteta22000
I don't understand this quote @bazsi : "For some of your log entries the name of the attachment is empty." . In my log dump, I have only 2 logs with attachmnts: attachment 'try.odt' and attachment 'house.html' ? Why do I get this: " "","","","","","","",try.odt,house.html,"","","","",house.html" ? With the context-lookup, I have duplicates: try.odt,house.html,house.html
Balazs Scheidler
@bazsi
let me check...
Balazs Scheidler
@bazsi
Ok, here's the thing: at aggregation time, a new synthetic message is created. This synthetic message is what gets emitted from the grouping-by() parser. So if you had 16 messages from the Ironport, you would have 17 messages in total: the original 16 + the one generated by syslog-ng. The goal of this synthetic message is to aggregate information from the previous 16 into just one. Since you have inherit-mode("context") in your config, this new message would contain a union of all name-value pairs in your original messages. And this is what you can extend with additional name-value pairs within your aggregate() option.
This means, that the synthetic message would have AGGR should-only-pass" name value pairs, since these are specified within aggregate().
Now, when the aggregation happens, the synthetic message becomes part of the context, so that your value() options can reference the aggregated name-value pairs. This is how your AGGR template can use ${mid_id}, this name-value pair is "inherited" from the context.
I hope it is clear up to this point.
With that all said, your synthetic message at the top of the context contains the ${attachment} field, inherited from any original message that we have seen so far, but it's going to contain the last of these values.
Balazs Scheidler
@bazsi
This means that even though you had only 2 attachments, "try.odt" and "house.html", your synthetic message would also contain $attachment and its value would be house.html, since that's the last message in the context.
$(context-lookup) iterates both the original messages and the synthetic one added on top, meaning house.html would be repeated.
I agree this is more complicated than it should be.
The solution to your immediate problem is to cut off the final element from the list returned by $(context-lookup), e.g. something like $(list-slice 0:-1 $(context-lookup ('$attachment' ne '') ${attachment}))
I'll check that expression in a minute.
Balazs Scheidler
@bazsi
The longer term solution is to make this a bit more intuitive. I was thinking on making $(context-lookup) ignore the last element of the context. The problem with this is that it is an incompatible change, one that is difficult to communicate and fix. The other problem with this approach is that only grouping-by() (and db-parser()) aggregation is what adds this extra, synthetic message in the context and there are other cases which don't have a synthetic message generated (e.g. simple matching rules in db-parser()).
Balazs Scheidler
@bazsi
Another solution would be to have an argument to $(context-lookup) and $(context-values) to ignore the synthetic message, which would at least make the expression a bit easier to come up with and read.
Balazs Scheidler
@bazsi
A 3rd solution is to simply document this better.
arteta22000
@arteta22000
Thanks @bazsi for the explanation.
I have another question. With these ugly cisco ironport logs. There are 3 different ids, MID is the main id. In the raw log example above, we see that a session can start with "ICID" ( New SMTP ICID 219986669 ...) and then another log does the mapping (ICID <> MID , Start MID 198090335 . ICID 219986669 ...).
Can I aggregate the logs with ICID and DCID and then aggregate with MID. I don't know if I'm clear?
arteta22000
@arteta22000
I repost an example of a log session:

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: New SMTP ICID 219986669 interface responsibility (8.16.8.16) address 17.39.24.2 reverse dns host cuis.com verified no

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: ICID 219986669 RELAY SG UNKNOWNLIST match sbrs[0.0:7.0] SBRS 5.1

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Start MID 198090335 ICID 219986669

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ICID 219986669 RID 0 To: fcvez@yaho.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ready 12710 bytes from tgamb@hotil.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 ICID 219986669 RID 0 To: fcvez@yaho.com

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 Subject 'Which add raise.'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 Message-ID 'tgamb@hotil.com'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 antivirus negative

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 attachment 'try.odt'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 attachment 'house.html'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Delivery start DCID 177209764 MID 198090335 to RID [0]

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 RID [0] Response '2.6.0 22633284F46CFCC33@mray.com [InternalId=4230] Queued mail for delivery'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 using engine: CASE spam positive

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: MID 198090335 rewritten to MID 204028130 by add-footer filter 'Footer Stamping'

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Message done DCID 158099468 MID 198090335 to RID [0] ['([168.xxxx.137])\r\n by hester.org with ESMTP; 11 Aug 1976 17:03:41 +0200)']

<22>Mar 18 10:21:29 server01/8.8.8.8 serv-mail1: Info: Message finished MID 198090335 done

Balazs Scheidler
@bazsi
This is possible to do in db-parser() but that's pretty arcane to use with its XML based format. @faxm0dem created a puppet wrapper for it and yaml based generation that is a lot easier to use.
The keyword to connect two independent correlation state is called the <create-context> action. Here's a bit of documentation for that:
Example: Creating a new context from an action

In syslog-ng OSE version 3.8 and newer, you can create a new context as an action. For details, see Element: create-context.

The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the <create-context> element.

<rule provider='test' id='12' class='violation'>
  <patterns>
    <pattern>simple-message-with-action-to-create-context</pattern>
  </patterns>
  <actions>
    <action trigger='match'>
      <create-context context-id='1000' context-timeout='60' context-scope='program'>
        <message inherit-properties='context'>
          <values>
            <value name='MESSAGE'>context message</value>
          </values>
        </message>
      </create-context>
    </action>
  </actions>
</rule>
In the case of db-parser() you are not limited to simply stuffing messages into a "context" that grouping-by() does, rather you have individual rules that can act based on the input.
This is a powerful feature but one that is pretty difficult to use. It honours your efforts with performance though.
What you would need to do basically is the following:
1) match the beginning of the session and correlate on ${CID}
Balazs Scheidler
@bazsi
2) match the "Start CID ... MID ..." message and on that create a context, this will create a NEW correlation context with a key that is present in the current message (e.g. ${MID}), in this case you can generate a synthetic message that becomes part of your new correlation state.
3) as you are matching the rest of the session (e.g. which contain ${MID}), you correlate against ${MID} and you will find a very-first synthetic message in this context. The one that you created as a result of the first point, e.g. the CID based correlation. Here you would need to represent all CID based data as a single message. With syslog-ng lists, you could encapsulate all CID based messages into a single syslog-ng list. e.g $(context-values $MSG) as a name-value pair. but if you have name-value pairs to extract, that might be a better solution.