Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Fabien Wernli
@faxm0dem
So yeah, I guess poke the doc team :)
Fabien Wernli
@faxm0dem
shall I open an issue on gh for that ?
László Várady
@MrAnno
yeah, I can only agree with you.
we should clearly state the type of LogMessage fields in the documentation. I'll open a ticket for the doc team
Fabien Wernli
@faxm0dem
oh, thanks, feel free to use my misleading example :)
László Várady
@MrAnno
Thanks. That is a really good one, actually :)
Fabien Wernli
@faxm0dem
thanks !
Alexander Kagno
@arkag
Alexander Kagno
@arkag
Some progress, updated the gist, now I'm getting systemd errors instead, which is good!
Alexander Kagno
@arkag
Ah, I was binding wrong.
László Szemere
@szemere
Hello @arkag , sorry for not responding earlier. Do I understand correctly that you have successfully solved the issue?
PLANTROON
@plantroon
I am trying to set up EWMM with default-network-drivers() source and syslog-ng() destination. https://gist.github.com/plantroon/1e4ea957a0d5021d8c274437d89127c0
the pasted setup produces this error: Invalid frame header; header=''
Attila Szakacs
@alltilla
@plantroon default-network-drivers() tries to guess the log format and parse accordingly. This error message might be caused by an invalid classification of the log format. I think it would be better to continue the investigation on GitHub's issues page (https://github.com/syslog-ng/syslog-ng/issues/new/choose). There are couple of ways to move forward, the easiest would be if you could send us the output of syslog-ng -Fedtv, or at least a couple more lines before the error line. Or if you can determine which incoming log message caused this error, and which port did it arrive to, we could try to replicate it ourselves.
PLANTROON
@plantroon
is there a way to forcibly use EWMM without default-network-drivers() ?
László Várady
@MrAnno
The ewmm() source can be used directly, until we find out what's wrong with your default-network-drivers() setup:
https://github.com/syslog-ng/syslog-ng/blob/c999529f5215cfeb238ff24d771807c968197b62/scl/ewmm/ewmm.conf#L92-L97
(syslog-ng has no syslog-ng() source, only a destination; and it is intentional, see the comment in the SCL snippet: https://github.com/syslog-ng/syslog-ng/blob/c999529f5215cfeb238ff24d771807c968197b62/scl/ewmm/ewmm.conf#L61-L78)
PLANTROON
@plantroon
I did not learn about the existence of ewmm() source from the manual, only now. So thanks, I will try with it first
László Várady
@MrAnno

Yeah, I think ewmm() as a source is not even documented, it's more of a building block for default-network-drivers().
ewmm-parser() and the $(format-ewmm) template function are the documented bits for custom EWMM-based source and destination-side processing:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.26/administration-guide/8#TOPIC-1430926

For general use, default-network-drivers() and the syslog-ng() destination are a good combination.

PLANTROON
@plantroon
yeah, I was adding EWMM to an existing system which processes logs in other ways already, so I couldn't simply use the default-network-drivers() and had to change ports and everything to avoid conflicts. But it didn't determine the source format correctly when coming from syslog-ng() destination over TLS. I may investigate it in the near future if I find the time.
Fabien Wernli
@faxm0dem
plantroon: so it's working now ?
PLANTROON
@plantroon
yes, it works perfectly. Thanks for the help
Fabien Wernli
@faxm0dem
\o/
Fabien Wernli
@faxm0dem
Hi, we're using a python parser, and it seems the callback retains state between executions
László Várady
@MrAnno

Hi @faxm0dem,
When your Python parser is referenced in a log path, an instance is created from the UpdateDiff class, so the fields of this object can be used to maintain a state.

I don't see non-local references in your example. What state are you referring to?

Fabien Wernli
@faxm0dem
thanks for the details
It seems there is some parts of log_message leaking from one message to the other
László Várady
@MrAnno
Oh, that sounds bad.
can this be reproduced steadily with the code you shared?
Fabien Wernli
@faxm0dem
We're trying
I'm checking if it's a PEBCAK and if not I'll make an easily reproducible test case
Fabien Wernli
@faxm0dem
Okay I've got a really weird behaviour
It seems to relate to an interaction between the rewrite rule and the python parser
The really weird part is that the rewrite is commented out in the log path, so it shouldn't even fire up
I even made it more impossible by adding an impossible condition :-)
Fabien Wernli
@faxm0dem
@MrAnno Hi, do you have some time to look into my problem today ?
László Várady
@MrAnno
Yeah, I'll try to look into it in the late afternoon. :)
László Várady
@MrAnno
We've discussed https://gist.github.com/faxm0dem/db314833ac95be805e5e59439a8055c0 privately, it seems we've found a bug in the LogMessage::keys() method of the Python binding. I will open an issue (or a pull request) soon.
Fabien Wernli
@faxm0dem
Are tcp connections being kept open between syslog-ng instances?
Attila Szakacs
@alltilla
We have a keep-alive(yes/no) option, which keeps the listener and the connections alive when you reload syslog-ng.
Is that what you are looking for? :)
sai
@saibug
hi community
Attila Szakacs
@alltilla
hi @saibug
sai
@saibug
i have a issue with syslog-ng !!
so , my architecture is :
2 syslog-ng (master/slave with failover)
each syslog-ng send writes to file log that is stored in NFS share :/
The question, is there any possibility to have only commun file for both syslog-ng ?
I know, it's not recommended, i have not tested to put any log file shipper in front, like filebeat just for merging both syslog-ng log file . you think it's possible ?
Attila Szakacs
@alltilla
@sai Is any other kind of log storing service available to you? Like ElasticSearch, MongoDB, Kafka, SQL, ...? syslog-ng can send logs to all of these, and they can handle multiple syslog-ng connections in the same time, so there won't be any resource allocation problems. Alternatively, as you said, you can write to different files on the NFS, and merge them with a script. I have found an old mailing list thread about this exact situation, it might help you a bit: https://lists.balabit.com/pipermail/syslog-ng/2005-January/006954.html
sai
@saibug

@sai Is any other kind of log storing service available to you? Like ElasticSearch, MongoDB, Kafka, SQL, ...? syslog-ng can send logs to all of these, and they can handle multiple syslog-ng connections in the same time, so there won't be any resource allocation problems. Alternatively, as you said, you can write to different files on the NFS, and merge them with a script. I have found an old mailing list thread about this exact situation, it might help you a bit: https://lists.balabit.com/pipermail/syslog-ng/2005-January/006954.html

thanks for the reply !!
for my use casse, i don't need to send syslog-ng to any other destination, these logs are used for nagios monitoring at this time

and there is check that needs only one input file
so , for this i need some advises :)
any other ideas ?
i have already tested to put filebeat in front, and it works