syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
- load balancing strategies
- acknowledgement process
- What if one program hangs? We do not change message orders, so it will block all other programs :(
2 replies
My assumption is, that someone probably want to keep sending messages via other processes, while one of them is blocked. (Not viable using round-robin balancing.)
(In the meantime: My earlier statement was incorrect, acknowledgement was not an issue in case of program destinations, since syslog-ng consider the message acknowledged if the write on the pipe was successfull.)
HOST, with a limit of 5
pre-fork(2)
do
rm ${NEWFILE}
done
This way once the file is read by syslog-ng it is getting deleted. here i am assuming that the file is process by syslog-ng ( which is the case for most of the time) I see all the files reaching my elasticsearch. Howerver I see below entries in log 2021-06-08T11:32:14.259750] Follow-mode file source not found, deferring open; filename='/var/log/apache2/93200/20210608/20210608-1702/20210608-170214-YL9VPvA4UO9fAg2IkuODWQAAAAI' is there any issue due to this ? i expect this will remove the file from monitoring ( hence keep the number of files well under the max-file count)
Hi @Homeshjoshi_twitter,
If I understand it correctly, the inotifywait -m -r -e close_writescript of yours will remove the file right after a process writing that file closes its fd.
Fortunately, when syslog-ng opens a file for reading, it keeps that file open. This means that if the file is deleted before it is fully processed by syslog-ng, we can still finish reading the file (The file will not really be deleted until the last FD is closed).
The only problematic scenario is when a file is deleted, but syslog-ng has not yet been able to open the file in that amount of time.
I think this is the case when you get the message you mentioned: Follow-mode file source not found
wildcard-file() source monitors files continuously, we don't close those files after processing them, so I think you either have to add a few-second delay before removing the file (in the hope syslog-ng can open it), or you should choose a different approach (a simple delay does not guarantee anything, but monitoring syslog-ng statistics (syslog-ng-ctl stats), or searching for internal debug logs that state we opened the file might actually work)
@Homeshjoshi_twitter
What I finally understand from your reply that my script is not going to work.
Adding a few-second delay might work, but it's an ugly hack, and nothing is guaranteed.
The better option (until we implement #3695) would be something that relies on the statistics of syslog-ng.
For example, it is possible to write a script that periodically checks the output of the sbin/syslog-ng-ctl query get 'src.file.*.processed' command, and removes the file only when the processed counter of the file can be found in the list, and it is greater than 0.
This should work reliably only if you do NOT re-create the same file (path) after deleting it (syslog-ng file statistics are not reset after the file is removed).
@arekm:matrix.org Hi,
The fallback flag is for processing messages that are not processed by any other normal log paths. You can not mark log_net as an "invisible" destination, but I'm pretty sure we can refactor/rephrase your configuration to achieve what you want; for example, using if-else blocks, or embedded log paths, final flags, or just filters.
Can you share all of your log paths, where s_sys is used? Exactly what messages would you like to see arrive into d_messages?
Now I'm seeing incorrect count values for suppressions messages on remote logs.
Please note that client is configured with suppress(600) and server is configured with suppress(0)
remote logs = server logs
when a message is for client, it starts suppression and looks for repeated messages.
once this repeated message steak is over, it logs "<MSG> repeated N times" and that log is also stored in server.
Now the problem I'm facing is the "repeated N times" value of N is sometimes different in server logs.
So Is there any open issue present currently? Do you know if this type of issue occurred before and fixed in later releases?
Hi @mshah618,
Now the problem I'm facing is the "repeated N times" value of N is sometimes different in server logs.
Could you elaborate on this a bit more? Did you find some inconsistencies between the client and server logs when using the suppress option?
Or do you expect a fixed amount of repetition within 600 seconds?
When a client is configured with the suppress(T) option, consecutive repetitive messages will be sent only once within that T timeframe. Please note that any non-identical message between 2 identical messages will reset the suppress counter.
After T seconds or when a non-identical message is found, the following summary will be sent to the server:Last message 'msg' repeated N times, suppressed by syslog-ng on ...
Can you share me your email id? I'll share those configs to you.
Could you share the relevant parts of your configuration publicly? More people can help you that way. :) You can remove sensitive information from the config.
If that's not possible, you can, of course, send your config to any of us in a private message.
Hi @mshah618,
Now the problem I'm facing is the "repeated N times" value of N is sometimes different in server logs.
Could you elaborate on this a bit more? Did you find some inconsistencies between the client and server logs when using the suppress option?
Or do you expect a fixed amount of repetition within 600 seconds?
I see inconsistencies between client and server logs when using suppress option.