Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 13:29
    k-128 starred syslog-ng/syslog-ng
  • 12:06
    MrAnno commented #4173
  • 12:02
    MrAnno commented #4209
  • 10:20
    HofiOne commented #4209
  • 10:20
    HofiOne commented #4209
  • 07:59
    mitzkia commented #4211
  • 03:50
    talltechy starred syslog-ng/syslog-ng
  • Dec 01 21:43
    MrAnno commented #4173
  • Dec 01 19:21
    bazsi ready_for_review #4211
  • Dec 01 19:20
    bazsi edited #4211
  • Dec 01 19:20
    bazsi commented #4211
  • Dec 01 19:18
    bazsi synchronize #4211
  • Dec 01 18:29
    kira-syslogng commented #4173
  • Dec 01 17:50
    MrAnno commented #4209
  • Dec 01 17:49
    MrAnno commented #4209
  • Dec 01 17:48
    MrAnno commented #3785
  • Dec 01 17:39
    MrAnno commented #4209
  • Dec 01 17:39
    MrAnno commented #4209
  • Dec 01 17:31
    bazsi synchronize #4173
  • Dec 01 16:40
    bazsi edited #4173
Attila Szakacs
@alltilla

Another query: I use the geoip parser but sometimes the IP is missing and this generates an error in the logs. Is there a way to make the parse conditional on there being something to process?

You can do something like this:

log { 
  source(s_src);
  if ( "${YOUR_IP_MACRO}" ne "" ) {
    parser(p_geoip);
  };
  destination(d_dest);
};
1 reply
Homesh
@Homeshjoshi_twitter

@alltilla currently I am getting the output as below

"python": {
"error_message[1]": {
"uri": "/",
"unique_id": "YI745wh9F2ssHXPZeiHwjQAAAAQ",
"tag": "Suspicious Unusual User Agent (python-requests). Disable this rule if you use python-requests/. ",
"severity": "CRITICAL",
"rev": "4",
"msg": "Suspicious User-Agent",
"line": "218",
"level": "",
"id": "332039",
"hostname": “1.1.1.1”,
"file": "/usr/share/sw/rules/20_sw_useragents.conf",
"client": "4.78.120.9"
},
"error_message[0]": {
"uri": "/",
"unique_id": "YI745wh9F2ssHXPZeiHwjQAAAAQ",
"tag": "Suspicious activity detected - Host header is a numeric IP address",
"severity": "NOTICE",
"rev": "4",
"msg": "Others",
"line": "89",
"level": "",
"id": "331032",
"hostname": “1.1.1.1”,
"file": "/usr/share/sw/rules/00_sw_zz_strict.conf",
"client": "4.78.120.9"
}
},

can you make it as an array of objects

"python":
"error_message": [
{
"uri": "/",
"unique_id": "YI745wh9F2ssHXPZeiHwjQAAAAQ",
"tag": "Suspicious Unusual User Agent (python-requests). Disable this rule if you use python-requests/. ",
"severity": "CRITICAL",
"rev": "4",
"msg": "Suspicious User-Agent",
"line": "218",
"level": "",
"id": "332039",
"hostname": “1.1.1.1”,
"file": "/usr/share/sw/rules/20_sw_useragents.conf",
"client": "4.78.120.9"
},
{
"uri": "/",
"unique_id": "YI745wh9F2ssHXPZeiHwjQAAAAQ",
"tag": "Suspicious activity detected - Host header is a numeric IP address",
"severity": "NOTICE",
"rev": "4",
"msg": "Others",
"line": "89",
"level": "",
"id": "331032",
"hostname": “1.1.1.1”,
"file": "/usr/share/sw/rules/00_sw_zz_strict.conf",
"client": "4.78.120.9"
}
]

Homesh
@Homeshjoshi_twitter
This will help me in kibana as it supports aggregation for single field e.g python.error_message.id I think you need to change your python code for this. Please help as I have no idea on python coding.
Russell Fulton
@rful011

The syslog source expects RFC6587 type framed logs. You should use a simple network in your case:

source s_eset {
    network( transport("tcp")  port(5514) keep-alive(yes));
    };

Doh! I was sure I had tried that! Sigh... Thanks!

Russell Fulton
@rful011
I have an application that is sending json in the MESSAGE. I assumed that I could parse this using the json parser but that seems to assume that the incoming stream is all json, not syslog with the Message in json format.
May  8 16:40:08 secmgrprd02 syslog-ng[3682]: json-parser(): failed to extract JSON members into name-value pairs. The parsed/extracted JSON payload was not an object; input='2021-05-08T04:40:08.374Z nodappprd01 ERAServer 3568 - - {"event_type":"FilteredWebsites_Event","ipv4":"172.24.45.167","hostname":"md378033.uoa.auckland.ac.nz","source_uuid":"a2ac336a-61a8-4a7d-8423-82756293da47","occured":"08-May-2021 04:24:26","severity":"Warning","event":"An attempt to connect to URL","target_address":"127.0.0.1","target_address_type":"IPv4","scanner_id":"HTTP filter","action_taken":"blocked","object_uri":"localhost.auckland.ac.nz","hash":"3A973AEF21BDDD57A32468471EFB577E15CDEB53","username":"UOA\\cnim002","processname":"C:\\Users\\cnim002\\AppData\\Local\\Mozilla Firefox\\firefox.exe","rule_id":"Website certificate revoked"}', extract_prefix='(null)'
Russell Fulton
@rful011
I tried passing "${MESSAGE}"to the parser but got syntax errors.
Yash Mathne
@yashmathne
Given that marker() function of the JSON parser is used for mixed messages, maybe you can append the message with a fixed marker and then use the marker() function to parse the JSON payload. (PS, fellow user, not associated with the team)
Russell Fulton
@rful011

Thanks Yash, your post prompted me to RTFM (again ;) and I see that it is supposed to parse the MESSAGE by default. So I am now doubly puzzled why this is failing.

hmmm... looking at the pcap (full message in previous message) we see
0x0060: 3220 2d20 2d20 efbb bf7b 2265 7665 6e74 2.-.-....{"event
I can't figure out what the non ascii efbbbf (less renders this as <U+FEFF>) is about but I now suspect that that is the cause of the problem. This is being sent by the eset application??? The is an option for length framing but I have that disabled. This does not look like a length!

I am now using a program destination with a template the includes only the MESSAGE macro. It gets the whole record!

My guess is that this chunk of non ascii at the start of the message is breaking syslog-ng's parsing of the message.

Presumably this is a bug in eset. Does anyone use syslog from eset?

Homesh
@Homeshjoshi_twitter

Thanks Yash, your post prompted me to RTFM (again ;) and I see that it is supposed to parse the MESSAGE by default. So I am now doubly puzzled why this is failing.

hmmm... looking at the pcap (full message in previous message) we see
0x0060: 3220 2d20 2d20 efbb bf7b 2265 7665 6e74 2.-.-....{"event
I can't figure out what the non ascii efbbbf (less renders this as <U+FEFF>) is about but I now suspect that that is the cause of the problem. This is being sent by the eset application??? The is an option for length framing but I have that disabled. This does not look like a length!

I am now using a program destination with a template the includes only the MESSAGE macro. It gets the whole record!

My guess is that this chunk of non ascii at the start of the message is breaking syslog-ng's parsing of the message.

Presumably this is a bug in eset. Does anyone use syslog from eset?

can you try with flags(no-parse) at the source {} refer https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.17/administration-guide/63 , from the admin guide "By default, syslog-ng OSE parses every message using the syslog-parser as a syslog message, and fills the macros with values of the message. The syslog-parser does not discard messages: the message cannot be parsed as a syslog message, the entire message (including its header) is stored in the $MSG macro. If you do not want to parse the message as a syslog message, use the flags(no-parse) option of the source."

Homesh
@Homeshjoshi_twitter
@alltilla did you get a chance to look at my issue regarding the python code to store "array of objects" and also earlier requirement for "Read mode" in wild card file. Thanks
Attila Szakacs
@alltilla

@alltilla did you get a chance to look at my issue regarding the python code to store "array of objects" and also earlier requirement for "Read mode" in wild card file. Thanks

Okay, so this is a bit hacky, because our format-json can only put strings in lists. However I managed to assemble the list in python, and told format-json in the destination template to handle that part as a literal string, so it did not add any additional formatting.

@version: 3.30

python {
import re
import json

class ErrorMessageParser(object):
  def init(self, options):
    self.prefix = options["prefix"] if "prefix" in options.keys() else ""
    return True

  def parse(self, msg):
    i = 0
    error_message_list = []

    while True:
      error_message_raw = msg["json.audit_data.error_messages[{}]".format(i)]
      if not error_message_raw:
        msg["error_message"] = json.dumps(error_message_list)
        return True

      error_message_entry = {}

      kv_pairs = re.findall(b"\[[^ ]+ [^\]]+\]", error_message_raw)
      for kv_pair in kv_pairs:
        delim_index = kv_pair.index(b" ")
        key = kv_pair[1:delim_index].decode()
        value = kv_pair[delim_index+2:-2].decode()
        error_message_entry[key] = value

      error_message_list.append(error_message_entry)

      i += 1
};

parser p_json {
  json-parser(prefix("json."));
  python(class("ErrorMessageParser") options("prefix", "python."));
};

source s_apache {
  file(
    "/tmp/in.json"
    follow-freq(60)
    flags(no-parse)
    log-fetch-limit(100)
  );
};

destination d_json {
  file(
    "/tmp/test.json"
    template("$(format-json --scope nv_pairs --pair uniqueId=\"${json.transaction.transaction_id}\" --pair error_message=literal(${error_message}))\n")
  );
};

log {
  source(s_apache);
  parser(p_json);
  destination(d_json);
};
Note that json and syslog-ng key-value pairs are mixed with this solution, if you are planning to send a message like this with a non format-json() templated destination, it will cause some confusion about the formatting.
Regarding the read-mode, we did not reach a conclusion yet, I will keep you posted.
Homesh
@Homeshjoshi_twitter
@alltilla Thanks a lot for all the help. keep me posted regarding the read mode.
Russell Fulton
@rful011
can someone please point me to some docs which spell out the support in the enterprise edition of syslog-ng for the various reliable transport standards. Is RLTP an implementation of RELP ? We have a several of apps that can use RELP.
1 reply
Yash Mathne
@yashmathne

Hey guys,
I have been part of this community for the past 2 months or so. I just received the news that my proposal for GSOC to work under this organisation has been accepted. I really look forward to furthering my engagement with this community and seeing this project through with everyone's concerns in mind. My proposal is up on the GSOC page of our Github, if you have any concerns or suggestions feel free to hit me up.

Looking forward to a fruitful summer!

Balazs Scheidler
@bazsi
Welcome Yash! Good luck for your project.
Homesh
@Homeshjoshi_twitter
I want to convert all the parsed field names to lowercase . e.g Host will become host, User-Agent will become user-agent. In the documentation there is is something like this mentioned rewrite rlower {
groupset ("$(lowercase '$
')" values(".json.request.headers"));
}; But this has no effect. can someone please help me ?
4 replies
Mark
@markfaine
Does anyone know if the log messages are processed by channels in parallel or in sequence (serially)? I'm not sure it makes a practical difference but I'd still like to know. The docs say that every channel receives every message in the junction, so that sounds like they are processed in parallel.
Balazs Scheidler
@bazsi
@markfaine the idea is that you shouldn't care and syslog-ng can decide how to best process messages in parallel branches of the pipeline. Today however they are processed in order. There are even some flags that when enabled will cause processing to be serialized, for instance flags(final) causes the subsequent branches not to be processed if one matches.
Mark
@markfaine

@bazsi This is exactly the issue I have. It seems that if I have something like:

channel {
...
flags('final');
}
channel {
...
flags('final');
}

Only the first one is ever checked, I know that not every message matches on the first channel filters. Yet, it checks the first one and if that doesn't work it doesn't seem to check any of the others.

Balazs Scheidler
@bazsi
are these channels within the same parent block?
Mark
@markfaine
Yes
Balazs Scheidler
@bazsi
Try enabling debug and trace, syslog-ng should log internal messages about the evaluation of the filters.
ehlo550
@ehlo550

Hi,
I have a little problem and cannot find an answer in the documentation for it.
I use a config similar to this:

network(ip(0.0.0.0) transport(tcp) port(1515) flags(store-raw-message));
log { source(s_aggregation); destination(d_splunk); };

Unfortunately I got a Cisco ISE sending syslog there and now I see that the messages are split on seperate files when I receive a message containing NULL values.

<181>May 18 22:36:22 ise01 CISE_RADIUS_Accounting .....cisco-av-pair=dc-certainty-metric=0, cisco-av-pair=dc-opaque=, cisco-av-pair=dc-protocol-map=1, .....

So in a pcap we can see that cisco-av-pair=dc-opaque=, is like


\x61\x69\x72\x3d\x64\x63\x2d\x6f\x70\x61\x71\x75\x65\x3d\x00\x00
\x00\x02\x00\x00\x00\x01\x00\x00\x00\x00\x2c\x20\x63\x69\x73\x63
--> 
air=dc-opaque=............, cisc

What is the best way to fix this?
Can you please help me?

13 replies
ehlo550
@ehlo550
image.png
Fabien Wernli
@faxm0dem
Hi folks !
Quick question : is it possible to multi-thread program() destinations ? e.g. fork multiple copies thereof ?
László Szemere
@szemere
Hello @faxm0dem , unfortunately it is not possible at the moment. How do you imagine this feature? (I mean from the point of pipes/communication flow. Currently it is a 1-1 connection between syslog-ng and the started process.)
Fabien Wernli
@faxm0dem
I was thinking about a mode where messages could be independantly handled
so multiple forks of program would be spawned, and messages load-balanced among them
László Szemere
@szemere
I was unable to find an already opened issue for this. I have some questions on top of my head, some can be discussed in a new issue ticket, some maybe here?
  • load balancing strategies
  • acknowledgement process
  • What if one program hangs? We do not change message orders, so it will block all other programs :(
I think this would be a convenient feature, however I am not sure if it is worth it to take this responsibility (from the point of Syslog-ng).
Maybe a "man in the middle" program can take care of this functions.
Homesh
@Homeshjoshi_twitter
@alltilla For this log sample {"transaction":{"time":"30/May/2021:19:57:39 +0530","transaction_id":"YLOg2-JSdAoTKJPtgtTaOQAAAAM","remote_address":"103.137.155.148","remote_port":9863,"local_address":"2.2.2.2","local_port":80},"request":{"request_line":"GET /?user=%3Cscript%3Ealert(123)%3C/script%3E HTTP/1.1","headers":{"Host":"www.abc.com","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:78.0) Gecko/20100101 Firefox/78.0","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8","Accept-Language":"en-US,en;q=0.5","Accept-Encoding":"gzip, deflate","DNT":"1","Connection":"keep-alive","Upgrade-Insecure-Requests":"1"}},"response":{"protocol":"HTTP/1.1","status":403,"headers":{"X-Content-Type-Options":"nosniff","Status":"403 Forbidden","Connection":"close","Content-Length":"1344","Content-Type":"text/html; charset=UTF-8"}},"audit_data":{"messages":["Access denied with code 403 (phase 2). Pattern match \"(?:< ?script|(?:<|< ?/)(?:(?:java|vb)script|about|applet|activex|chrome|qx?ss|embed)|< ?/?i?frame\\b|< ?img src ?=|< ?base href ?=)\" at REQUEST_URI. [file \"/usr/share/sitewall/rules/10_asl_rules.conf\"] [line \"1085\"] [id \"340147\"] [rev \"146\"] [msg \"XSS\"] [data \"<script\"] [severity \"CRITICAL\"] [tag \"Potential Cross Site Scripting Attack\"]"],"error_messages":["[file \"apache2_util.c\"] [line 273] [level 3] [client 103.137.155.148] ModSecurity: Access denied with code 403 (phase 2). Pattern match \"(?:< ?script|(?:<|< ?/)(?:(?:java|vb)script|about|applet|activex|chrome|qx?ss|embed)|< ?/?i?frame\\\\\\\\b|< ?img src ?=|< ?base href ?=)\" at REQUEST_URI. [file \"/usr/share/sitewall/rules/10_asl_rules.conf\"] [line \"1085\"] [id \"340147\"] [rev \"146\"] [msg \"XSS\"] [data \"<script\"] [severity \"CRITICAL\"] [tag \"Potential Cross Site Scripting Attack\"] [hostname \"www.abc.com\"] [uri \"/\"] [unique_id \"YLOg2-JSdAoTKJPtgtTaOQAAAAM\"]"],"action":{"intercepted":true,"phase":2,"message":"Pattern match \"(?:< ?script|(?:<|< ?/)(?:(?:java|vb)script|about|applet|activex|chrome|qx?ss|embed)|< ?/?i?frame\\b|< ?img src ?=|< ?base href ?=)\" at REQUEST_URI."},"handler":"application/x-httpd-php","stopwatch":{"p1":514,"p2":2116,"p3":0,"p4":0,"p5":61,"sr":80,"sw":0,"l":0,"gc":0},"producer":["ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/)","201903261539"],"server":"Apache/2.4.29 (Ubuntu)","engine_mode":"ENABLED"}} I am getting incorrect value for parsing of field as ""client": "03.137.155.14" it should be "client": "103.137.155.14"
2 replies
Fabien Wernli
@faxm0dem
If one is blocking, it's not syslog-ng's problem ?
László Szemere
@szemere

My assumption is, that someone probably want to keep sending messages via other processes, while one of them is blocked. (Not viable using round-robin balancing.)

(In the meantime: My earlier statement was incorrect, acknowledgement was not an issue in case of program destinations, since syslog-ng consider the message acknowledged if the write on the pipe was successfull.)

Fabien Wernli
@faxm0dem
That being said, it could probably make sense to add the round-robin functionality externally as part of a configuration keyword so that it could be used for multiple destinations:
log {
  load-balance(key('HOST'), forks(5), destination(d_myprogram));
};
would be neat, huh ?
László Szemere
@szemere
I like it! @alltilla what do you think?
Fabien Wernli
@faxm0dem
or maybe max-forks(5)
so that it spawns as many forks as there are HOST, with a limit of 5
maybe pre-fork(2)
you get the idea :-D
László Szemere
@szemere
Sorry for not answering, we are discussing your idea IRL with @alltilla . We have some interesting ideas. (Thinking about a more general solution, so not only program destination will benefit from it.)
László Szemere
@szemere
I open a github issue for the topic, where we can track the upcoming ideas.
Peter Czanik
@czanik
Yeah, sending logs over multiple TCP connections would also cool
like recent changes in the MongoDB destination made significant speed ups even without bulk mode
László Szemere
@szemere
The mentioned GitHub issue: syslog-ng/syslog-ng#3692 please feel free to comment on it, if I missed something.
Fabien Wernli
@faxm0dem
awesome, thanks !