Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 17:09
    czanik commented #4060
  • 16:46
    MrAnno commented #4060
  • 16:36
    MrAnno commented #4060
  • 16:35
    MrAnno commented #4060
  • 16:31
    MrAnno commented #4060
  • 16:31
    MrAnno commented #4060
  • 16:30
    MrAnno commented #4060
  • 16:25
    MrAnno commented #4060
  • 16:10
    MrAnno closed #4070
  • 16:10
    MrAnno commented #4070
  • 15:25
    impishskald commented #4070
  • 15:20
    MrAnno labeled #4070
  • 15:17
    MrAnno commented #4070
  • 15:07
    impishskald commented #4070
  • 14:58
    MrAnno commented #4070
  • 14:52
    MrAnno commented #4070
  • 14:29
    impishskald commented #4070
  • 14:25
    impishskald commented #4070
  • 14:21
    MrAnno commented #4070
  • 14:06
    impishskald opened #4070
arekm
@arekm:matrix.org
[m]
The goal here is to use default syslog-ng config and only put one file in syslog.d/ that will push all logs to remote server, too. if/else/final won't play well with such assumption
(one file in... == one additional configuration file)
László Várady
@MrAnno
Oh I see. Is this your own custom default config? (I checked our default configurations (DEB, RPM, Arch Linux packaging) and could not find this fallback path.)
arekm
@arekm:matrix.org
[m]
PLD distro config.. which I could change in pld itself, too actually. I'll see how syslog-ng upstream config looks like
mshah618
@mshah618
Hi All,
I'm new Here and want to explain an issue I'm facing with syslog-ng.
I'm communicating to a server through a client. The client is on syslog-ng-v3.16.1 and the server is on syslog-ng-v3.10.1
Now I'm seeing incorrect count values for suppressions messages on remote logs.
Please note that client is configured with suppress(600) and server is configured with suppress(0)
any suggestion is appreciable.
Balázs Barkó
@Barkodcz

Hi @mshah618 ,
what do you mean it's incorrect count values for suppression messages?
just to be clear, when you say remote logs you mean the server logs?

For my opinion, it is possible if you update syslog-ng on your server it will solve the problem.

mshah618
@mshah618
Thanks, @Barkodcz for responding.
remote logs = server logs
when a message is for client, it starts suppression and looks for repeated messages.
once this repeated message steak is over, it logs "<MSG> repeated N times" and that log is also stored in server.
Now the problem I'm facing is the "repeated N times" value of N is sometimes different in server logs.
Hope I explain the issue well.
So Is there any open issue present currently? Do you know if this type of issue occurred before and fixed in later releases?
Balázs Barkó
@Barkodcz
Hi @mshah618 , can you send me your config please?
It could help for debugging and understanding the problem.
mshah618
@mshah618
Hi @Barkodcz , Can you share me your email id? I'll share those configs to you.
László Várady
@MrAnno

Hi @mshah618,

Now the problem I'm facing is the "repeated N times" value of N is sometimes different in server logs.

Could you elaborate on this a bit more? Did you find some inconsistencies between the client and server logs when using the suppress option?
Or do you expect a fixed amount of repetition within 600 seconds?

When a client is configured with the suppress(T) option, consecutive repetitive messages will be sent only once within that T timeframe. Please note that any non-identical message between 2 identical messages will reset the suppress counter.
After T seconds or when a non-identical message is found, the following summary will be sent to the server:
Last message 'msg' repeated N times, suppressed by syslog-ng on ...

Can you share me your email id? I'll share those configs to you.

Could you share the relevant parts of your configuration publicly? More people can help you that way. :) You can remove sensitive information from the config.
If that's not possible, you can, of course, send your config to any of us in a private message.

mshah618
@mshah618

Hi @mshah618,

Now the problem I'm facing is the "repeated N times" value of N is sometimes different in server logs.

Could you elaborate on this a bit more? Did you find some inconsistencies between the client and server logs when using the suppress option?
Or do you expect a fixed amount of repetition within 600 seconds?

I see inconsistencies between client and server logs when using suppress option.

László Várady
@MrAnno
What exactly do you see? When suppressing logs on the client side, the server receives the summary message itself (Last message 'msg' repeated N times), so messages are not recounted on the server side, you will have to see the exact same message there.
mshah618
@mshah618
yes but I see different counts sometimes
not only on server side, but sometimes I see wrong count number on client side and correct count on server side

Some times server log contains wrong number of repeated messages, some times client log contains it.

Some times client and server logs show correct, equal value.

László Várady
@MrAnno
How do you validate which side has the "correct count"?
Do you have 2 different destinations (a local file and a network?) configured with the suppress() option on the client side?
mshah618
@mshah618
Yes
László Várady
@MrAnno
Every destination has its own suppress timer and counter, so the two will not always produce the same output.
For example, if one of the 2 destinations receives messages from other sources as well, the suppress output will be completely different.
László Várady
@MrAnno

Yeah, that might be the consequence of 2 destinations receiving messages from slightly different sources or log paths.
Could you share your client and server configs, please? You can send it as a private message here on Gitter, if you want to.

I'll summarize what we've found here, publicly.

mshah618
@mshah618
Sure
László Várady
@MrAnno
A brief summary: we could not figure out how the client forwards messages to the server because the configuration does not contain any network destinations, only files.
I think some infrastructural details should be shared before moving forward with this problem. TLDR: suppress() is done on the client-side (by destinations), each destination has its own suppress counter/timer, so inconsistencies between the server and client side seems to be a configuration issue (for example, 2 different destinations are used on slightly different log paths, etc).
PLANTROON
@plantroon
I have some information in PROGRAM macro, which I would like to use in the destination's path on the filesystem. However, I want to use only part of PROGRAM in the path, how do I go about this?
PLANTROON
@plantroon

ok I figured it out. If I have it like this:

log {
    source(s_myservers_tls);
    filter{program(apache_error)};
    parser(p_apache_error);
    rewrite(r_rewrite_subst);
    destination(df_new_apache_error);
};

will the rewrite be applied AFTER the previous filter() directive? (so will the filter() be respected? Because with rewrite being done, the filter's input data would change and not work for this given filter)

PLANTROON
@plantroon
what happens is that apache_error_domain.tld changes to domain_tld with that rewrite
László Szemere
@szemere

Hello @plantroon ,

the statements in the Syslog-ng configuration file are "executed" from top down. So the filter will be applied to the original PROGRAM field. And the later rewrite will only affect the following destination statement.

However I am not sure about your "p_apache_error" parser. If your parser also process/alters the PROGRAM field, you might want to put it prior your filter statement.

PLANTROON
@plantroon

Thanks :)

However I am not sure about your "p_apache_error" parser. If your parser also process/alters the PROGRAM field, you might want to put it prior your filter statement.

if I understand correctly, only the message itself is available to the parser, no? Therefore it does not work with the program field... at least mine does not. I use ewmm syslog driver if that has anything to do with it.

László Szemere
@szemere

First things first, I think your setup will work as is.

A little bit of explanation, why your parser only receives the message part:
Every parser has a "template" option, with a default $MESSAGE value. The template option defines the input of the parser. Since evmm already parsed the incoming message as a apache log, your PROGRAM field, and other apache related macros are already filled.

(Usually if somebody writes a custom parser, they use "flags(no-parse) on the source side, which combined with the default MESSAGE value for the template, will result getting EVERYTHING)

PLANTROON
@plantroon
ah I see :) thanks for explaining. It appears to work fine, it's been like this since 23:00 yesterday and logs are parsed and saved correctly
engineer11111
@engineer11111

hey there !

i'm trying to send logs from syslog-ng to elasticsearch(basically using syslog-ng as a proxy) and while this config perfectly works on my ubuntu 20.04 when i try to launch it inside docker it doesn't send anything

syslog-ng.conf

@version: 3.31
@include "scl.conf"

source s_network {
  network(
    port(514)
    transport("tcp")
  );
};

destination d_es{
  elasticsearch-http(
    index("syslog-ng")
    url("https://")
    user("admin")
    password("password")
    type("")
    tls( peer-verify(no) )
  );
};
log {
  source(s_network);
  destination(d_es);
};

things i've tried already :

  • launching container with --privileged flag
  • enabling verbose via syslog-ng-ctl mode - barely any new messages

i'm using docker image 3.31.2-buster aka latest

could anyone please help me with that one ?

László Szemere
@szemere

Hello! Naive question: are you sure that your container/syslog-ng indeed receives any log messages? (Exposing the 514 port, or using host network)

I am thinking about the Syslog-ng debug messages with the beginning of the "Incoming log entry" text.
i.e.: [2021-07-21T16:41:59.091937] Incoming log entry; line='<165>1 2003-10-11T22:14:15.003Z 1.2.3.4 evntslog - ID47 [example123456789

engineer11111
@engineer11111

hi!
thank you for getting back to me

it does receive log entries, i've tried both with simple network requests and setting it up to collect logs from syslog + making logs entries via logger

do i enable verbose mode the right way?

root@6b6a564571aa:/# syslog-ng-ctl verbose --set=on
VERBOSE=1
engineer11111
@engineer11111

ah, turned out verbose != debug
after enabling debug mode i've got a bunch of messages like this one

curl: HTTP response received; url='https://1234.eu-central-1.es.amazonaws.com/_bulk', status_code='200', body_size='276', batch_size='1', redirected='0', total_time='0.075', worker_index='1', driver='d_es#0', location='#buffer:4:3'

index still isn't there unfortunately

László Szemere
@szemere
May I recommend turning on trace messages too? (./syslog-ng-ctl trace --set=on) - Warning: it will print out a LOT of information.
With trace enabled cURL print out the response body too in the debug logs. Maybe the response will tell us more about the problem.
engineer11111
@engineer11111

i did it yesterday actually and indeed it produces a lot of messages
some of them like these

Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''
Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''
Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''
Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''
Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''

what logs specifically should i look for since it's really a lot of them ?

László Szemere
@szemere

Sorry, I did not take time to bring up an elastic instance. So I hit a local python wevserver with a simple http destination. I got the following output:

2021-07-23T07:33:42.837203] cURL debug; worker='0', type='data_in', data='<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN".        "http://www.w3.org/TR/html4/strict.dtd">.<html>.    <head>.        <meta http-
equiv="Content-Type" content="text/html;charset=utf-8">.        <title>Error response</title>.    </head>.    <body>.        <h1>Error response</h1>.        <p>Error code: 501</p>.        <p>Message: Unsupported
 method (\'POST\').</p>.        <p>Error code explanation: HTTPStatus.NOT_IMPLEMENTED - Server does not support this operation.</p>.    </body>.</html>.'

But there should be something similar in your output too.

engineer11111
@engineer11111
traces are fine, i think i found the issue

oh, no worries

here are the logs btw

Jul 23 08:18:01 01ee2ee74e14 syslog-ng[1]: cURL debug; worker='0', type='header_out', data='POST /_bulk HTTP/2..Host: 1234.eu-central-1.es.amazonaws.com..authorization: Basic YWW4xQDM=..user-agent: syslog-ng 3.25.1/libcurl 7.68.0..accept: */*..content-type: application/x-ndjson..content-length: 348....'
Jul 23 08:18:01 01ee2ee74e14 syslog-ng[1]: cURL debug; worker='0', type='ssl_data_out', data='....}'
Jul 23 08:18:01 01ee2ee74e14 syslog-ng[1]: cURL debug; worker='0', type='data_out', data='{"index":{"_index":"RELEASE-NAME-syslog-ng"}}.{"PROGRAM":"syslog-ng","PRIORITY":"debug","PID":"1","MESSAGE":"Sending HTTP request; url=\'https://1234.eu-central-1.es.amazonaws.com/_bulk\'","ISODATE":"2021-07-23T08:18:01+00:00","HOST":"01ee2ee74e14","FACILITY":"syslog","@timestamp":"2021-07-23T08:18:01+00:00"}.'
Jul 23 08:18:01 01ee2ee74e14 syslog-ng[1]: cURL debug; worker='0', type='text', data='We are completely uploaded and fine.'
so the issue was my ignorance actually : elasticsearch index cannot start with a capital letter https://discuss.elastic.co/t/why-cant-an-index-start-with-an-uppercase-letter/72435/3
László Szemere
@szemere
Good catch!
engineer11111
@engineer11111
really thank you for your time and help!
not a syslog-ng issue at all but well, maybe it will help someone in the future
skappen
@skappen

We have observed a syslog-ng(version is 3.16.1) hang issue at customer side. It is happens occassionly.Board resets due to watchdog also happens after this.
They have been logging some test messages using "logger" application. Whenever they see logger gets stuck then a syslog-ng core dump is generated by manually sending an
ABRT signal. Syslog-ng cpu usage is zero and it is in S state when the issue hits. Given below is the back trace.
Back-trace showing that syslog-ng is sitting at epoll_wait, waiting for an fd event to get triggered. Is there any known issue with "ivykis" library in syslog-ng?


Core was generated by `/usr/sbin/syslog-ng --process-mode background -C /var/lib/syslog-ng -p /var/run'.
Program terminated with signal 6, Aborted.

#0 0x0000003ff10e8f73 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:81

81 ../sysdeps/unix/syscall-template.S: No such file or directory.

#0 0x0000003ff10e8f73 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:81

No locals.

#1 0x0000003ff3c85595 in iv_fd_epoll_timerfd_poll (st=0xa01320, active=0x7ffd0a9513b0, abs=0x0) at iv_fd_epoll.c:436

run_timers = 0
batch = 0x7ffd0a9512b0
ret = <optimized out>
run_events = <optimized out>
i = <optimized out>

#2 0x0000003ff3c82eb7 in iv_fd_poll_and_run (st=st@entry=0xa01320, abs=<optimized out>) at iv_fd.c:202

active = {next = 0x7ffd0a9513b0, prev = 0x7ffd0a9513b0}
run_timers = <optimized out>

#3 0x0000003ff3c83d2a in iv_main () at iv_main_posix.c:112

_abs = {tv_sec = 0, tv_nsec = 0}
abs = <optimized out>
st = 0xa01320
run_timers = <optimized out>

#4 0x0000003ff3c49e96 in main_loop_run (self=0x3ff3ec3840 <main_loop>) at lib/mainloop.c:573

No locals.

#5 0x00000000004018e9 in main (argc=1, argv=0x7ffd0a951558) at syslog-ng/main.c:307

rc = <optimized out>
ctx = 0x9fee50
error = 0x0
main_loop = 0x3ff3ec3840 <main_loop>

exit_before_main_loop_run = 0

Kókai Péter
@Kokan
At first I would not suspect ivykis. I'd advise to upgrade if possible as likely this issues is already resolved (for example in 3.17.1 there was a fix regarding window size: syslog-ng/syslog-ng#2167 that looks familiar)
skappen
@skappen
Thanks. Upgrading syslog-ng is not an active plan. Can we narrow it down the issue further or is there any workaround like setting the syslog-ng configuration parametrs log-iw-size(), log-fetch-limit() would help?
Kókai Péter
@Kokan
if you are really facing the issue I've linked, in your case only option is to patch port or upgrade.
skappen
@skappen
So for backporting fix your suggestion is to back-port these patches https://github.com/syslog-ng/syslog-ng/pull/2167/commits right?