Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 09:15
    bazsi synchronize #4158
  • 09:12
    github-actions[bot] commented #4158
  • 09:12
    bazsi opened #4158
  • 08:57
    MrAnno review_requested #4143
  • 07:57
    bazsi closed #4157
  • 07:52
    bazsi synchronize #4143
  • 07:48
    bazsi synchronize #4143
  • 05:05
    Niba-nazar commented #4157
  • Sep 28 13:20
    ryanfaircloth commented #4133
  • Sep 28 13:15
    ryanfaircloth commented #4155
  • Sep 28 13:14
    ryanfaircloth commented #4155
  • Sep 28 13:12
    bazsi commented #4157
  • Sep 28 13:03
    bazsi commented #4155
  • Sep 28 13:02
    bazsi commented #4133
  • Sep 28 12:18
    ryanfaircloth commented #4133
  • Sep 28 11:28
    ryanfaircloth commented #4155
  • Sep 28 11:06
    Niba-nazar opened #4157
  • Sep 28 11:06
    Niba-nazar labeled #4157
  • Sep 28 10:24
    bazsi commented #4155
  • Sep 28 09:40
    AliasgarSabunwala starred syslog-ng/syslog-ng
Mark
@markfaine

@bazsi This is exactly the issue I have. It seems that if I have something like:

channel {
...
flags('final');
}
channel {
...
flags('final');
}

Only the first one is ever checked, I know that not every message matches on the first channel filters. Yet, it checks the first one and if that doesn't work it doesn't seem to check any of the others.

Balazs Scheidler
@bazsi
are these channels within the same parent block?
Mark
@markfaine
Yes
Balazs Scheidler
@bazsi
Try enabling debug and trace, syslog-ng should log internal messages about the evaluation of the filters.
ehlo550
@ehlo550

Hi,
I have a little problem and cannot find an answer in the documentation for it.
I use a config similar to this:

network(ip(0.0.0.0) transport(tcp) port(1515) flags(store-raw-message));
log { source(s_aggregation); destination(d_splunk); };

Unfortunately I got a Cisco ISE sending syslog there and now I see that the messages are split on seperate files when I receive a message containing NULL values.

<181>May 18 22:36:22 ise01 CISE_RADIUS_Accounting .....cisco-av-pair=dc-certainty-metric=0, cisco-av-pair=dc-opaque=, cisco-av-pair=dc-protocol-map=1, .....

So in a pcap we can see that cisco-av-pair=dc-opaque=, is like


\x61\x69\x72\x3d\x64\x63\x2d\x6f\x70\x61\x71\x75\x65\x3d\x00\x00
\x00\x02\x00\x00\x00\x01\x00\x00\x00\x00\x2c\x20\x63\x69\x73\x63
--> 
air=dc-opaque=............, cisc

What is the best way to fix this?
Can you please help me?

13 replies
ehlo550
@ehlo550
image.png
Fabien Wernli
@faxm0dem
Hi folks !
Quick question : is it possible to multi-thread program() destinations ? e.g. fork multiple copies thereof ?
László Szemere
@szemere
Hello @faxm0dem , unfortunately it is not possible at the moment. How do you imagine this feature? (I mean from the point of pipes/communication flow. Currently it is a 1-1 connection between syslog-ng and the started process.)
Fabien Wernli
@faxm0dem
I was thinking about a mode where messages could be independantly handled
so multiple forks of program would be spawned, and messages load-balanced among them
László Szemere
@szemere
I was unable to find an already opened issue for this. I have some questions on top of my head, some can be discussed in a new issue ticket, some maybe here?
  • load balancing strategies
  • acknowledgement process
  • What if one program hangs? We do not change message orders, so it will block all other programs :(
I think this would be a convenient feature, however I am not sure if it is worth it to take this responsibility (from the point of Syslog-ng).
Maybe a "man in the middle" program can take care of this functions.
Homesh
@Homeshjoshi_twitter
@alltilla For this log sample {"transaction":{"time":"30/May/2021:19:57:39 +0530","transaction_id":"YLOg2-JSdAoTKJPtgtTaOQAAAAM","remote_address":"103.137.155.148","remote_port":9863,"local_address":"2.2.2.2","local_port":80},"request":{"request_line":"GET /?user=%3Cscript%3Ealert(123)%3C/script%3E HTTP/1.1","headers":{"Host":"www.abc.com","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:78.0) Gecko/20100101 Firefox/78.0","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8","Accept-Language":"en-US,en;q=0.5","Accept-Encoding":"gzip, deflate","DNT":"1","Connection":"keep-alive","Upgrade-Insecure-Requests":"1"}},"response":{"protocol":"HTTP/1.1","status":403,"headers":{"X-Content-Type-Options":"nosniff","Status":"403 Forbidden","Connection":"close","Content-Length":"1344","Content-Type":"text/html; charset=UTF-8"}},"audit_data":{"messages":["Access denied with code 403 (phase 2). Pattern match \"(?:< ?script|(?:<|< ?/)(?:(?:java|vb)script|about|applet|activex|chrome|qx?ss|embed)|< ?/?i?frame\\b|< ?img src ?=|< ?base href ?=)\" at REQUEST_URI. [file \"/usr/share/sitewall/rules/10_asl_rules.conf\"] [line \"1085\"] [id \"340147\"] [rev \"146\"] [msg \"XSS\"] [data \"<script\"] [severity \"CRITICAL\"] [tag \"Potential Cross Site Scripting Attack\"]"],"error_messages":["[file \"apache2_util.c\"] [line 273] [level 3] [client 103.137.155.148] ModSecurity: Access denied with code 403 (phase 2). Pattern match \"(?:< ?script|(?:<|< ?/)(?:(?:java|vb)script|about|applet|activex|chrome|qx?ss|embed)|< ?/?i?frame\\\\\\\\b|< ?img src ?=|< ?base href ?=)\" at REQUEST_URI. [file \"/usr/share/sitewall/rules/10_asl_rules.conf\"] [line \"1085\"] [id \"340147\"] [rev \"146\"] [msg \"XSS\"] [data \"<script\"] [severity \"CRITICAL\"] [tag \"Potential Cross Site Scripting Attack\"] [hostname \"www.abc.com\"] [uri \"/\"] [unique_id \"YLOg2-JSdAoTKJPtgtTaOQAAAAM\"]"],"action":{"intercepted":true,"phase":2,"message":"Pattern match \"(?:< ?script|(?:<|< ?/)(?:(?:java|vb)script|about|applet|activex|chrome|qx?ss|embed)|< ?/?i?frame\\b|< ?img src ?=|< ?base href ?=)\" at REQUEST_URI."},"handler":"application/x-httpd-php","stopwatch":{"p1":514,"p2":2116,"p3":0,"p4":0,"p5":61,"sr":80,"sw":0,"l":0,"gc":0},"producer":["ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/)","201903261539"],"server":"Apache/2.4.29 (Ubuntu)","engine_mode":"ENABLED"}} I am getting incorrect value for parsing of field as ""client": "03.137.155.14" it should be "client": "103.137.155.14"
2 replies
Fabien Wernli
@faxm0dem
If one is blocking, it's not syslog-ng's problem ?
László Szemere
@szemere

My assumption is, that someone probably want to keep sending messages via other processes, while one of them is blocked. (Not viable using round-robin balancing.)

(In the meantime: My earlier statement was incorrect, acknowledgement was not an issue in case of program destinations, since syslog-ng consider the message acknowledged if the write on the pipe was successfull.)

Fabien Wernli
@faxm0dem
That being said, it could probably make sense to add the round-robin functionality externally as part of a configuration keyword so that it could be used for multiple destinations:
log {
  load-balance(key('HOST'), forks(5), destination(d_myprogram));
};
would be neat, huh ?
László Szemere
@szemere
I like it! @alltilla what do you think?
Fabien Wernli
@faxm0dem
or maybe max-forks(5)
so that it spawns as many forks as there are HOST, with a limit of 5
maybe pre-fork(2)
you get the idea :-D
László Szemere
@szemere
Sorry for not answering, we are discussing your idea IRL with @alltilla . We have some interesting ideas. (Thinking about a more general solution, so not only program destination will benefit from it.)
László Szemere
@szemere
I open a github issue for the topic, where we can track the upcoming ideas.
Peter Czanik
@czanik
Yeah, sending logs over multiple TCP connections would also cool
like recent changes in the MongoDB destination made significant speed ups even without bulk mode
László Szemere
@szemere
The mentioned GitHub issue: syslog-ng/syslog-ng#3692 please feel free to comment on it, if I missed something.
Fabien Wernli
@faxm0dem
awesome, thanks !
Homesh
@Homeshjoshi_twitter
Hi @alltilla, do you have any update for me regarding my requirement of "logstash read mode" kind of feature in syslog-ng, the workaround of using script ( I tested script ) is not very efficient as for every file I have to check with syslog-ng-ctl and for a file which is not yet process by syslog (e.g condition when max-file limit reached) syslog-ng-ctl returns no information so I have to further check if syslog-ng is running. This method is not efficient when you have thousands of logs to process, BTW i am very happy with the syslog-ng performance compared to logstash, syslog-ng is consuming almost nothing (40 MB) memory compared to that of logstash it is 1GB , also since syslog-ng is written is c cpu usage is normal
Attila Szakacs
@alltilla
Hi @Homesh, Thanks for reminding me! We had 1 meeting, where we talked about this feature request, but we did not reach a conclusion. To make it more transparent, and to make sure that it does not get forgotten, I have made a GitHub issue for it: syslog-ng/syslog-ng#3695
Homesh
@Homeshjoshi_twitter
Thanks @alltilla
Homesh
@Homeshjoshi_twitter
I am working on another workaround. Idea is since only the syslog-ng will access the files, I am using inotifywait -m -r -e close_write --format '%w%f' "${MONITORDIR}" | while read NEWFILE
do
rm ${NEWFILE}
done
This way once the file is read by syslog-ng it is getting deleted. here i am assuming that the file is process by syslog-ng ( which is the case for most of the time) I see all the files reaching my elasticsearch. Howerver I see below entries in log 2021-06-08T11:32:14.259750] Follow-mode file source not found, deferring open; filename='/var/log/apache2/93200/20210608/20210608-1702/20210608-170214-YL9VPvA4UO9fAg2IkuODWQAAAAI' is there any issue due to this ? i expect this will remove the file from monitoring ( hence keep the number of files well under the max-file count)
Second I see empty directories (due to above mentioned script) e.g /var/log/apache2/93200/20210608/20210608-1702 will this cause issue for syslog-ng ( as it will still monitor these directories for possible new file, I think I should also delete the empty directories immediately. Please suggest
Homesh
@Homeshjoshi_twitter
Hi @alltilla can you please suggest.
László Várady
@MrAnno

Hi @Homeshjoshi_twitter,

If I understand it correctly, the inotifywait -m -r -e close_writescript of yours will remove the file right after a process writing that file closes its fd.

Fortunately, when syslog-ng opens a file for reading, it keeps that file open. This means that if the file is deleted before it is fully processed by syslog-ng, we can still finish reading the file (The file will not really be deleted until the last FD is closed).

The only problematic scenario is when a file is deleted, but syslog-ng has not yet been able to open the file in that amount of time.
I think this is the case when you get the message you mentioned: Follow-mode file source not found

László Várady
@MrAnno
The wildcard-file() source monitors files continuously, we don't close those files after processing them, so I think you either have to add a few-second delay before removing the file (in the hope syslog-ng can open it), or you should choose a different approach (a simple delay does not guarantee anything, but monitoring syslog-ng statistics (syslog-ng-ctl stats), or searching for internal debug logs that state we opened the file might actually work)
Homesh
@Homeshjoshi_twitter
Hi @MrAnno , thanks for the reply. My logic is once the file is read by syslog-ng it should get deleted, since no new log is going to be written by application in that file (due to concurrent logging). What I finally understand from your reply that my script is not going to work. I can not keep debug ON for production to check for files process by syslog-ng. My only option left is to use elasticsearch's filebeat as it can delete the file once it is process like logstash ( filebeat is lighter than that of logstash) and then filebeat will write the logs to a new single file or it can forward to syslog-ng on syslog port (e.g 514), I do not wanted to add dependency on filebeat or any other program, but now I don't have option until syslog-ng will offer the same feature of filebeat or logstash. Logstash hangs after 7 to 8 days and consumes lot of memory and cpu. That is why I want to replac logstash with syslog-ng. Thanks again for your reply.
László Várady
@MrAnno

@Homeshjoshi_twitter

What I finally understand from your reply that my script is not going to work.

Adding a few-second delay might work, but it's an ugly hack, and nothing is guaranteed.

The better option (until we implement #3695) would be something that relies on the statistics of syslog-ng.
For example, it is possible to write a script that periodically checks the output of the sbin/syslog-ng-ctl query get 'src.file.*.processed' command, and removes the file only when the processed counter of the file can be found in the list, and it is greater than 0.

This should work reliably only if you do NOT re-create the same file (path) after deleting it (syslog-ng file statistics are not reset after the file is removed).

Homesh
@Homeshjoshi_twitter
Interesting !! let me try this and share my experience. Thanks a lot @MrAnno
László Várady
@MrAnno
My pleasure
arekm
@arekm:matrix.org
[m]
Hi, using log { source(s_sys); destination(log_net); }; to log all logs into remote syslog. But that makes "log { source(s_sys); destination(d_messages); flags(fallback); };" not logging things locally due to fallback flag. Is there a way to mark my log_net logging somehow, so it doesn't interfere with fallback logging?
László Várady
@MrAnno

@arekm:matrix.org Hi,
The fallback flag is for processing messages that are not processed by any other normal log paths. You can not mark log_net as an "invisible" destination, but I'm pretty sure we can refactor/rephrase your configuration to achieve what you want; for example, using if-else blocks, or embedded log paths, final flags, or just filters.

Can you share all of your log paths, where s_sys is used? Exactly what messages would you like to see arrive into d_messages?

arekm
@arekm:matrix.org
[m]
The goal here is to use default syslog-ng config and only put one file in syslog.d/ that will push all logs to remote server, too. if/else/final won't play well with such assumption
(one file in... == one additional configuration file)
László Várady
@MrAnno
Oh I see. Is this your own custom default config? (I checked our default configurations (DEB, RPM, Arch Linux packaging) and could not find this fallback path.)
arekm
@arekm:matrix.org
[m]
PLD distro config.. which I could change in pld itself, too actually. I'll see how syslog-ng upstream config looks like
mshah618
@mshah618
Hi All,
I'm new Here and want to explain an issue I'm facing with syslog-ng.