Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 10:47
    Indra2108 starred syslog-ng/syslog-ng
  • 10:47
    Indra2108 starred syslog-ng/syslog-ng
  • 10:37
    Kokan commented #3853
  • 10:35
    Barkodcz edited #3853
  • 09:18
    calcky starred syslog-ng/syslog-ng
  • 07:02
    owl4ce starred syslog-ng/syslog-ng
  • Dec 01 20:43
    Wernfried commented #3475
  • Dec 01 20:42
    Wernfried closed #3475
  • Dec 01 15:42
    kira-syslogng commented #3853
  • Dec 01 15:00
    Barkodcz opened #3853
  • Dec 01 13:57
    faxm0dem labeled #3852
  • Dec 01 13:57
    faxm0dem opened #3852
  • Nov 30 11:16
    lbudai commented #3844
  • Nov 29 19:12
    bazsi commented #3844
  • Nov 29 19:05
    kira-syslogng commented #3844
  • Nov 29 18:48
    bazsi commented #3844
  • Nov 29 18:47
    bazsi synchronize #3844
  • Nov 29 18:45
    bazsi synchronize #3844
  • Nov 29 18:40
    kira-syslogng commented #3844
skappen
@skappen
Hi @szemere Many thanks. The issue is also reproduced on syslog-ng 3.19.1. I am trying to test other versions between 3.19.1 and 3.24.1 but at present I dont have any live system with versions. I need to build other syslog-ng versions first to verify it. Upgrade is not a plan as it may break other stuffs. So looking to figure out the patch and backport the same.
skappen
@skappen
Hi @szemere I have narrowed i down it further and identified that issue is not seen on syslog-ng-3.23.1 after the commits https://github.com/syslog-ng/syslog-ng/pull/2798/commits. Looks like default time-reap value of 60 seconds in prior versions to syslog-ng-3.23.1 causes this behaviour.
László Szemere
@szemere

Helllo @skappen , Thank you for reporting it back. Based on the linked PR, I figured out that in case when the suppress time is bigger than time_reap, the suppress message is lost.

So the problem still exists, the PR only hides it in your case, by disabling the reap timer when it is not configured explicitly.

I would like to discuss the topic with my colleagues. This looks like an undefined situation for me, at this moment I don't know what could be the expected behavior.

1 reply
Fabien Wernli
@faxm0dem
Hi there, just added this to my base config because I didn't like how docker containers (--log-driver journald) ended up logging using their ID as PROGRAM:
source s_system {
  channel {
    source { system(); };
    # make sure docker containers don't get their PROGRAM set to their CONTAINER_ID
    # dockerd shall be their PROGRAM
    junction {
      channel {
        filter {
          "${.journald.SYSLOG_IDENTIFIER}" != ""
            and
          "${.journald.SYSLOG_IDENTIFIER}" == "${.journald.CONTAINER_ID}"
        };
        rewrite {
          set("${.journald._COMM}", value("PROGRAM"));
        };
      };
      channel {
        flags(fallback);
      };
    };
  };
};
I'm mentioning it here, as it might help others, or why not make it into the s_system SCL
Peter Czanik
@czanik
Thank you!
Balazs Scheidler
@bazsi
did anyone notice that dockerhub is not building our images anymore?
László Várady
@MrAnno

@bazsi
hmm, we wanted to try GitHub's own docker registry anyway (https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry), which seems to be less limited, faster, and currently free. We could use this for dbld images, if it turnes out to be good for us; and in case we want to keep the main syslog-ng image on Docker Hub (balabit/syslog-ng), we could "reverse" the trigger direction between our CI and Docker Hub for that image (by building the image on GitHub, and then pushing it to Docker Hub from GitHub Actions).

subscribing to a Docker Hub Pro plan (or applying for Docker Open Source) would not be a big deal either, but dockerhub has been quite unstable for the past year, so I think the GH container registry might be worth a quick look.

what do you think?

László Várady
@MrAnno
(In the meantime, I filled the "Docker Open Source Community Application" for syslog-ng.)
Balazs Scheidler
@bazsi
I agree, that we should try an alternative and we are running our actions in github anyway, I guess this would be much faster.
László Várady
@MrAnno
Thanks for noticing about the dockerhub build!
I just finished filling out the dockerhub open source application form, we'll see what happens next week :)
Balazs Scheidler
@bazsi
I tried to build one of the dbld images locally and it failed. That's why I have checked.
Sanjay Patel
@San_j_ay_twitter
hello
has anyone get syslog-ng kafka to work?
I cannot figure out how to load kafka module.
Sanjay Patel
@San_j_ay_twitter
I cant find syslog-ng-cfg-db.py file on my server either
Balazs Scheidler
@bazsi
Which version do you have?
Are you sure you have kafka in your build?
Sanjay Patel
@San_j_ay_twitter
@bazsi I have 3.31
how do I verify if kafka is in the build?
Balazs Scheidler
@bazsi
Run syslog-ng --module-registry
Sanjay Patel
@San_j_ay_twitter
not in there
that is my question, how do I add kafka-c to my current install?
Balazs Scheidler
@bazsi
Which platform?
Robert Paschedag
@rpasche

Hi everyone....we are looking for alternatives to logstash and want to test syslog-ng. We are using a lot of filters and grok to - sometimes - create "nested objects" (within elasticsearch). Can someone show me a simple example, howto extract information from within a syslog message (${MESSAGE}) field, so we are able to "create" a new field? Example: Syslog message contains VM some_super_shiny_vm_name has been started. I want to add a new - nested - field like vm.name (JSON: {"vm": {"name": "my_super_shiny_vm_name"}}with the name of the VM ("my_super_shiny_vm_name"), so I could create a json object like

$(format-json message=${MESSAGE} vm.name=${my_new_extracted_field})

Thanks you for help.

6 replies
Joep van Diessen
@joepvand
Hello everyone, i currently have syslog-ng, but i want a way where a custom webinterface can see logs in realtime. does anyone know where to start with this?
2 replies
Sanjay Patel
@San_j_ay_twitter
I cant get syslog-ng to accept messages, when I do tcpdump I an see port open and messages coming in but when I run syslog-ng it does not accept the messages.
Sanjay Patel
@San_j_ay_twitter
I can see the port open but when I do syslog-ng -Fevd it shows no data
László Várady
@MrAnno
@San_j_ay_twitter Could you share the output of syslog-ng -Fevd?
Sanjay Patel
@San_j_ay_twitter
how do I send without flooding?
64 replies
Sanjay Patel
@San_j_ay_twitter
still stuck
tcpdump shows packets coming on udp but syslog-ng not picking them up
László Várady
@MrAnno

Hi,

If I understood your last reply correctly, netcat does not receive messages/packets either.
In case that's true, your issue is not really a syslog-ng configuration issue, it must be something environmental.

Sanjay Patel
@San_j_ay_twitter
@MrAnno it was a IPtable issue. Even with firewall off Ubuntu firewall rules apply. I added this to make it work. iptables -I INPUT -p udp -j ACCEPT. Adding it here incase someone else runs into this issue.
Robert Paschedag
@rpasche
Short question: Is the last pattern .*some kind of "workaround" within the new regexp-parserpattern so a message does not get discarded in case none of the previous pattern finds anything? Because I'm currently testing this new regexp-parser and added several patterns to extract information and noticed, that my message was never written to the destination. This seems to be caused by the fact, that none of my patterns really matched something. So is some kind of last pattern like .* the only chance I do not lose a message or are there flags that I can use to prevent messages to be discarded? Thank you for your help
16 replies
Robert Paschedag
@rpasche
Can you help me once more? I want to convert the syslog-ng TAGS into a JSON array, that should then be used within elasticsearch. Currently, the TAGS are set, but set a a static string (comma separated). I tried it with some rewrite but currently I cannot get it work. So I tried for example
rewrite {
  set("$(explode ',' ${TAGS})", value("tags"));
};
Balazs Scheidler
@bazsi
Just use the list type hint
1 reply
Fabien Wernli
@faxm0dem
rpasche: FWIW you could also use an Elasticsearch ingest pipeline : https://www.elastic.co/guide/en/elasticsearch/reference/7.15/split-processor.html
2 replies
Fabien Wernli
@faxm0dem
we like to do most in syslog-ng too, but when we have json upstream, we don't want syslog-ng to mess up its types, so we use ingest pipelines to parse the json in ES
just for the record: I understand your use-case
ann-lang
@ann-lang

Hello,
I am an assistant professor at Beihang University. Our team is doing a study about GSoC mentors, aiming to understand the motivations, challenges, strategies, and gains of GSoC mentors. To this end, we designed a questionnaire. We sincerely invite GSoC mentors to participate in this survey. Your feedback is very important for us.

Questionnaire link: https://forms.gle/rgAWwmrvrCb5XdAq9

If you are interested in this study, welcome to join our follow-up interview! Thank you very much!

Sincerely,

Xin Tan

2 replies
kiphackman
@kiphackman
hi guys im getting an issue on syslog-ng version 3.23.1 where logs are not written to the disk. I've spent a lot of time troubleshooting but had no luck. Would love some insight or assistance from the community if you guys may have any recommendations. Sorry for the trouble. And thank you so much
6 replies
Wojciech Adam Koszek
@wkoszek_gitlab
Imagine you're building a hobby project. You don't want to invest in Logstash'es and Kibanas. You love syslog-ng however. What's the easiest way to get some dashboard for logs?
To give you an example: modern tool to get web stats could be Grafana or something, but one can use Webalizer tool that has 1995 look, but does the job. I wonder if there's some 1995 tool that just works and gives me simple querability without all the maintenance
2 replies
Fabien Wernli
@faxm0dem
I'm guessing grafana/loki would be the most straightforward route nowadays
Peter Czanik
@czanik
if you do not mind closed source and do not need scalability in the long run, I liked SEQ, when I tested it a while ago: https://www.syslog-ng.com/community/b/blog/posts/creating-a-new-http--based-syslog-ng-destination-seq
1 reply
Fabien Wernli
@faxm0dem_twitter
Hi I'm getting a json parsing error that I don't understand: https://gist.github.com/faxm0dem/463d632d2f4031d8236f3294a04a98f6
I'm guessing it's the NaN it doesn't like
Fabien Wernli
@faxm0dem_twitter
yeah that was it