Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 28 20:56
    MrAnno milestoned #3893
  • Jan 28 20:56
    MrAnno milestoned #3891
  • Jan 28 20:56
    MrAnno milestoned #3844
  • Jan 28 20:55
    MrAnno milestoned #3710
  • Jan 28 20:01
    MrAnno commented #3867
  • Jan 28 19:09
    philippem commented #3820
  • Jan 28 18:40
    philippem commented #3820
  • Jan 28 18:40
    ctrlbrk42 starred syslog-ng/syslog-ng
  • Jan 28 18:36
    MrAnno review_requested #3887
  • Jan 28 18:36
    MrAnno unassigned #3887
  • Jan 28 18:00
    kira-syslogng commented #3891
  • Jan 28 17:55
    kira-syslogng commented #3893
  • Jan 28 17:41

    MrAnno on master

    mainloop-worker: take care of i… mainloop-worker: drop WorkerOpt… mainloop-worker: rename worker … and 26 more (compare)

  • Jan 28 17:41
    MrAnno closed #3879
  • Jan 28 17:41
    MrAnno closed #3867
  • Jan 28 17:38
    kira-syslogng commented #3891
  • Jan 28 17:37
    bazsi synchronize #3891
  • Jan 28 17:33
    bazsi synchronize #3893
  • Jan 28 17:30
    MrAnno commented #3893
  • Jan 28 17:30
    MrAnno commented #3893
PLANTROON
@plantroon
ah I see :) thanks for explaining. It appears to work fine, it's been like this since 23:00 yesterday and logs are parsed and saved correctly
engineer11111
@engineer11111

hey there !

i'm trying to send logs from syslog-ng to elasticsearch(basically using syslog-ng as a proxy) and while this config perfectly works on my ubuntu 20.04 when i try to launch it inside docker it doesn't send anything

syslog-ng.conf

@version: 3.31
@include "scl.conf"

source s_network {
  network(
    port(514)
    transport("tcp")
  );
};

destination d_es{
  elasticsearch-http(
    index("syslog-ng")
    url("https://")
    user("admin")
    password("password")
    type("")
    tls( peer-verify(no) )
  );
};
log {
  source(s_network);
  destination(d_es);
};

things i've tried already :

  • launching container with --privileged flag
  • enabling verbose via syslog-ng-ctl mode - barely any new messages

i'm using docker image 3.31.2-buster aka latest

could anyone please help me with that one ?

László Szemere
@szemere

Hello! Naive question: are you sure that your container/syslog-ng indeed receives any log messages? (Exposing the 514 port, or using host network)

I am thinking about the Syslog-ng debug messages with the beginning of the "Incoming log entry" text.
i.e.: [2021-07-21T16:41:59.091937] Incoming log entry; line='<165>1 2003-10-11T22:14:15.003Z 1.2.3.4 evntslog - ID47 [example123456789

engineer11111
@engineer11111

hi!
thank you for getting back to me

it does receive log entries, i've tried both with simple network requests and setting it up to collect logs from syslog + making logs entries via logger

do i enable verbose mode the right way?

root@6b6a564571aa:/# syslog-ng-ctl verbose --set=on
VERBOSE=1
engineer11111
@engineer11111

ah, turned out verbose != debug
after enabling debug mode i've got a bunch of messages like this one

curl: HTTP response received; url='https://1234.eu-central-1.es.amazonaws.com/_bulk', status_code='200', body_size='276', batch_size='1', redirected='0', total_time='0.075', worker_index='1', driver='d_es#0', location='#buffer:4:3'

index still isn't there unfortunately

László Szemere
@szemere
May I recommend turning on trace messages too? (./syslog-ng-ctl trace --set=on) - Warning: it will print out a LOT of information.
With trace enabled cURL print out the response body too in the debug logs. Maybe the response will tell us more about the problem.
engineer11111
@engineer11111

i did it yesterday actually and indeed it produces a lot of messages
some of them like these

Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''
Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''
Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''
Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''
Jul 22 18:49:38 f62b8106c0bd syslog-ng[1]: cURL debug; worker='2', type='data_out', data=''

what logs specifically should i look for since it's really a lot of them ?

László Szemere
@szemere

Sorry, I did not take time to bring up an elastic instance. So I hit a local python wevserver with a simple http destination. I got the following output:

2021-07-23T07:33:42.837203] cURL debug; worker='0', type='data_in', data='<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN".        "http://www.w3.org/TR/html4/strict.dtd">.<html>.    <head>.        <meta http-
equiv="Content-Type" content="text/html;charset=utf-8">.        <title>Error response</title>.    </head>.    <body>.        <h1>Error response</h1>.        <p>Error code: 501</p>.        <p>Message: Unsupported
 method (\'POST\').</p>.        <p>Error code explanation: HTTPStatus.NOT_IMPLEMENTED - Server does not support this operation.</p>.    </body>.</html>.'

But there should be something similar in your output too.

engineer11111
@engineer11111
traces are fine, i think i found the issue

oh, no worries

here are the logs btw

Jul 23 08:18:01 01ee2ee74e14 syslog-ng[1]: cURL debug; worker='0', type='header_out', data='POST /_bulk HTTP/2..Host: 1234.eu-central-1.es.amazonaws.com..authorization: Basic YWW4xQDM=..user-agent: syslog-ng 3.25.1/libcurl 7.68.0..accept: */*..content-type: application/x-ndjson..content-length: 348....'
Jul 23 08:18:01 01ee2ee74e14 syslog-ng[1]: cURL debug; worker='0', type='ssl_data_out', data='....}'
Jul 23 08:18:01 01ee2ee74e14 syslog-ng[1]: cURL debug; worker='0', type='data_out', data='{"index":{"_index":"RELEASE-NAME-syslog-ng"}}.{"PROGRAM":"syslog-ng","PRIORITY":"debug","PID":"1","MESSAGE":"Sending HTTP request; url=\'https://1234.eu-central-1.es.amazonaws.com/_bulk\'","ISODATE":"2021-07-23T08:18:01+00:00","HOST":"01ee2ee74e14","FACILITY":"syslog","@timestamp":"2021-07-23T08:18:01+00:00"}.'
Jul 23 08:18:01 01ee2ee74e14 syslog-ng[1]: cURL debug; worker='0', type='text', data='We are completely uploaded and fine.'
so the issue was my ignorance actually : elasticsearch index cannot start with a capital letter https://discuss.elastic.co/t/why-cant-an-index-start-with-an-uppercase-letter/72435/3
László Szemere
@szemere
Good catch!
engineer11111
@engineer11111
really thank you for your time and help!
not a syslog-ng issue at all but well, maybe it will help someone in the future
skappen
@skappen

We have observed a syslog-ng(version is 3.16.1) hang issue at customer side. It is happens occassionly.Board resets due to watchdog also happens after this.
They have been logging some test messages using "logger" application. Whenever they see logger gets stuck then a syslog-ng core dump is generated by manually sending an
ABRT signal. Syslog-ng cpu usage is zero and it is in S state when the issue hits. Given below is the back trace.
Back-trace showing that syslog-ng is sitting at epoll_wait, waiting for an fd event to get triggered. Is there any known issue with "ivykis" library in syslog-ng?


Core was generated by `/usr/sbin/syslog-ng --process-mode background -C /var/lib/syslog-ng -p /var/run'.
Program terminated with signal 6, Aborted.

#0 0x0000003ff10e8f73 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:81

81 ../sysdeps/unix/syscall-template.S: No such file or directory.

#0 0x0000003ff10e8f73 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:81

No locals.

#1 0x0000003ff3c85595 in iv_fd_epoll_timerfd_poll (st=0xa01320, active=0x7ffd0a9513b0, abs=0x0) at iv_fd_epoll.c:436

run_timers = 0
batch = 0x7ffd0a9512b0
ret = <optimized out>
run_events = <optimized out>
i = <optimized out>

#2 0x0000003ff3c82eb7 in iv_fd_poll_and_run (st=st@entry=0xa01320, abs=<optimized out>) at iv_fd.c:202

active = {next = 0x7ffd0a9513b0, prev = 0x7ffd0a9513b0}
run_timers = <optimized out>

#3 0x0000003ff3c83d2a in iv_main () at iv_main_posix.c:112

_abs = {tv_sec = 0, tv_nsec = 0}
abs = <optimized out>
st = 0xa01320
run_timers = <optimized out>

#4 0x0000003ff3c49e96 in main_loop_run (self=0x3ff3ec3840 <main_loop>) at lib/mainloop.c:573

No locals.

#5 0x00000000004018e9 in main (argc=1, argv=0x7ffd0a951558) at syslog-ng/main.c:307

rc = <optimized out>
ctx = 0x9fee50
error = 0x0
main_loop = 0x3ff3ec3840 <main_loop>

exit_before_main_loop_run = 0

Kókai Péter
@Kokan
At first I would not suspect ivykis. I'd advise to upgrade if possible as likely this issues is already resolved (for example in 3.17.1 there was a fix regarding window size: syslog-ng/syslog-ng#2167 that looks familiar)
skappen
@skappen
Thanks. Upgrading syslog-ng is not an active plan. Can we narrow it down the issue further or is there any workaround like setting the syslog-ng configuration parametrs log-iw-size(), log-fetch-limit() would help?
Kókai Péter
@Kokan
if you are really facing the issue I've linked, in your case only option is to patch port or upgrade.
skappen
@skappen
So for backporting fix your suggestion is to back-port these patches https://github.com/syslog-ng/syslog-ng/pull/2167/commits right?
Kókai Péter
@Kokan
yes
skappen
@skappen
thanks for the help.
László Várady
@MrAnno
@skappen Please consider backporting #2589 as well, it fixes an ordering issue that may cause a similar hang.
László Várady
@MrAnno
@skappen Could you share your config too? logger may get stuck when flags(flow-control) is enabled on a log path where the system() source is used. flow-control stops the corresponding source if a destination on the same log path can not send messages out for some reason (network issue, slow disk, etc.)
2 replies
Homesh
@Homeshjoshi_twitter

@Homeshjoshi_twitter

What I finally understand from your reply that my script is not going to work.

Adding a few-second delay might work, but it's an ugly hack, and nothing is guaranteed.

The better option (until we implement #3695) would be something that relies on the statistics of syslog-ng.
For example, it is possible to write a script that periodically checks the output of the sbin/syslog-ng-ctl query get 'src.file.*.processed' command, and removes the file only when the processed counter of the file can be found in the list, and it is greater than 0.

This should work reliably only if you do NOT re-create the same file (path) after deleting it (syslog-ng file statistics are not reset after the file is removed).

This seems to be working for me. Thanks again for help

László Várady
@MrAnno
@Homeshjoshi_twitter My pleasure. Thanks for the feedback.
Homesh
@Homeshjoshi_twitter

I am running syslog-ng docker as

docker run -it --rm -v /syslog-ng/conf/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf -v /syslog-ng/conf.d:/etc/syslog-ng/conf.d -v /syslog-ng/GeoLite2-City.mmdb:/var/log/GeoLite2-City.mmdb --name syslog-ng1 balabit/syslog-ng:latest -ev

my config is

source s_10000 {
wildcard-file(
base-dir("/var/log/apache2/10000")
filename-pattern("*")
recursive(yes)
follow-freq(1)
flags(no-parse)
max-files(1000)
log-fetch-limit(200)
log-iw-size(200001)
);
};

log {
source(s_10000);
parser(p_json);
parser(p_geoip2);
rewrite(r_geoip2);

destination(d_json);

destination(d_100);
};

I am getting below error

2021-08-02T17:57:23.231455] directory-monitor-inotify: could not create inotify object; errno='Too many open files (24)'
[2021-08-02T17:57:23.231461] Wildcard source: could not create directory monitoring object! Possible message loss; dir='/var/log/apache2/10000/ssl/20210801/20210801-1726', location='/etc/syslog-ng/conf.d/10000.conf:2:2'

Basically there are multiple directories in /var/log/apache2/10000

e.g
./20210801
./20210801/20210801-1324
./20210801/20210801-1324/20090822-132420-SojdH8AAQEAAAugAQAAAAAA
./20210801/20210801-1324/20090822-132420-SojdH8AAQEAAAugAQEAAAAA

Should I increase the max-file say 2000 ?
László Várady
@MrAnno
@Homeshjoshi_twitter Too many open files (24) means you reached your max open fd limit.
You can adjust that limit with ulimit, for example.
Homesh
@Homeshjoshi_twitter
that is on host system or for docker ?
László Várady
@MrAnno
I'm not sure. Does docker run --ulimit nofile=10000:10000 make any difference? :)
Homesh
@Homeshjoshi_twitter
ho it does not make any difference,i already try that with --ulimit nofile=5000:5000
László Várady
@MrAnno
Can you execute the following in your docker container, please?
$ sysctl fs.inotify.max_user_instances
Homesh
@Homeshjoshi_twitter
ok
docker exec syslog-ng1 sysctl fs.inotify.max_user_instances
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"sysctl\": executable file not found in $PATH": unknown
root@612266594d0c:/# sysctl fs.inotify.max_user_instances
bash: sysctl: command not found
Homesh
@Homeshjoshi_twitter

I'm not sure. Does docker run --ulimit nofile=10000:10000 make any difference? :)

i removed all other and kept only 1000.conf and started docker with --ulimit nofile=10000:10000

syslog-ng docker container started successfully
Homesh
@Homeshjoshi_twitter
however if i add one more log source I again face the same error
2021-08-02T19:08:30.904599] directory-monitor-inotify: could not create inotify object; errno='Too many open files (24)'
[2021-08-02T19:08:30.904607] Wildcard source: could not create directory monitoring object! Possible message loss; dir='/var/log/apache2/93800/20210802/20210802-0618', location='/etc/syslog-ng/conf.d/93800.conf:2:2'
[2021-08-02T19:08:30.9046
even with docker run --ulimit nofile=60000:60000 I get the same error
László Szemere
@szemere
Hello @Homeshjoshi_twitter ,
just to clarify things. Can you test the ulimit nofile option with a "regular" container? (i.e. ubuntu:latest, or something) To make sure that you do not have any preferences locally which prevents docker from increasing the number of allowed file descriptors?
I tried --ulimit nofile=100:100 locally with syslog-ng and with other contianers and I was able to reproduce your "errno='Too many open files (24)'" issue. (And fix it by increasing ulimit.)
Homesh
@Homeshjoshi_twitter
Hi @szemere , Thanks for response. I am going to try the solution mentioned here "https://medium.com/@ivanermilov/how-to-fix-inotify-cannot-be-used-reverting-to-polling-too-many-open-files-bb1c1437dbf" So I am going to increase the limit on host system as for docker container it seems it share the setting of host, as mentioned here "https://stackoverflow.com/questions/57220658/change-inotify-max-user-instances-limit-in-docker-container" I will update my findings.
Homesh
@Homeshjoshi_twitter
looks like this works for me. Thanks you @MrAnno and @szemere for your help.
László Szemere
@szemere
Great news! Thank You for the feedback!
Homesh
@Homeshjoshi_twitter
Yesterday I capture these logs while I started the syslog-ng container is there any issue with this ?
2021-08-02T18:57:19.123744] Unknown argument, adding it to VARARGS; argument='persist_name', value='"d100"', reference='/etc/syslog-ng/conf.d/100.conf:2:5'
[2021-08-02T18:57:19.133280] Unknown argument, adding it to VARARGS; argument='persist_name', value='"d104"', reference='/etc/syslog-ng/conf.d/104.conf:2:5'
[2021-08-02T18:57:19.134027] Unknown argument, adding it to VARARGS; argument='persist_name', value='"d932"', reference='/etc/syslog-ng/conf.d/932.conf:2:5'
[2021-08-02T18:57:19.135120] Unknown argument, adding it to VARARGS; argument='persist_name', value='"d938"', reference='/etc/syslog-ng/conf.d/938.conf:2:5'
I have to create destination for each log source with logs forwarding to same elasticsearch server hence I am adding e.g persist-name("d100")
László Szemere
@szemere

Hello @Homeshjoshi_twitter , those logs are completely normal.

In more details: They are generated during grammar parsing. In your case they are referring to this line: https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elastic-http.conf#L36

In case of blocks there are named parameters like "index" in case of elasticsearch. And "everything else", alias "VARARGS". The later one is useful because it allows us to pass extra parameters to the underlining http destination, without the need to blindly repeat those options in elastic. As a result reducing the size of the glue code.

Homesh
@Homeshjoshi_twitter
Thanks for the brief answer @szemere
skappen
@skappen

On syslog-ng 3.16.1 suppress option in syslog-ng.conf configured as suppress(600).
When we send identical messages to a destination, message suppression does not work properly.
It logs the same sent message at an interval of 60 seconds.

sending message using logger as
logger -p debug "test messages..."

Aug 10 02:46:40 qafarm2-50 root: test messages...
Aug 10 02:47:40 qafarm2-50 root: test messages...
Aug 10 02:48:40 qafarm2-50 root: test messages...

It does not log "root[24697]: Last message 'test messages...' repeated N times, suppressed by syslog-ng".

When we configure syslog-ng 3.16.1 with option suppress(60) it works as expected.
We have tested syslog-ng version 3.24.1 where issue is not seen, the suppress(600) is working as expected.
Is it a known issue in syslog-ng 3.16.1 also could you help to pin point which commit fixed this issue?