Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 06:18
    bazsi commented #4027
  • May 27 10:42
    gaborznagy commented #3934
  • May 27 10:14
    kira-syslogng commented #3934
  • May 27 09:49
    gaborznagy commented #3934
  • May 27 06:24
    houzy starred syslog-ng/syslog-ng
  • May 26 14:08
    kira-syslogng commented #4015
  • May 26 13:46
    bazsi commented #4015
  • May 26 13:45
    bazsi synchronize #4015
  • May 26 13:22
    mohitvaid commented #4013
  • May 26 10:55
    shikharvashistha review_requested #4009
  • May 26 08:53
    Roffild closed #4020
  • May 26 08:53
    Roffild commented #4020
  • May 26 08:37
    Roffild labeled #4027
  • May 26 08:37
    Roffild opened #4027
  • May 26 08:21
    Roffild edited #4026
  • May 26 08:19
    Roffild labeled #4026
  • May 26 08:19
    Roffild opened #4026
  • May 25 18:11
    kira-syslogng commented #4025
  • May 25 17:50
    kira-syslogng commented #4024
  • May 25 17:50
    bazsi edited #4025
Balazs Scheidler
@bazsi
This doesn't sound like a flow control issue, then. You've set log-msg-size() to a pretty large value, how long messages do you have?
I haven't seen anything like this.
Can you send me a sample of input that reproduces this?
arteta22000
@arteta22000
@bazsi do you see any solution with my issue ?
Artem Panasenkov
@artem.panasenkov_gitlab

@bazsi sample message: { "krb_tuple": 972254290, "ip_svr_addr": "1.2.3.4", "ip_svr_port": 445, "ip_cli_addr": "4.3.2.1", "ip_cli_port": 50461, "ip_direction": "client2server", "hbcnt": 13238, "somecnt": 5918, "krb": { "pvno": 5, "msg_type": "ap_req", "ap_options": { "use_session_key": false, "mutual_required": true }, "ticket": { "tkt_vno": 5, "realm": "EXAMPLE.COM", "sname": { "name_type": "NT_SRV_INST", "name_string": [ "cifs", "host.example.com" ] }, "enc_part": { "etype": "AES256_CTS_HMAC_SHA1_96", "kvno": 11, "cipherhash": "bca4d09f545483c262dfe026f81081f21938bd6eac66616a9db07d34d35cda04" } }, "authenticator": { "etype": "AES256_CTS_HMAC_SHA1_96", "cipherhash": "6e10a7f9e0aefe9e07fc70d7753d26069d59c382b2521b4a1f8c98534279e1a9" } }, "timestamp": 1645737961423, "unique_id": "7a7de91f350-d859-4392-b14c-a177eb50", "run_session": "6de0d2fa-96dd-4d1c-85c3-516e79cc2242", "kdc_status": 4, "src_ip": "4.3.2.1", "src_hostname": "host.example.com", "agent_version": "1.2.3.4" }

average message size is 1kb, occaisional 10kb, those are less frequent than 1 in a million and don't appear to be correlated and get through fine

28 replies
Artem Panasenkov
@artem.panasenkov_gitlab
we have some more data, much fewer messages are lost when we increased tcp stack buffer on host OS
clay584
@clay584
Given this config, I should only see syslog messages on the destination where the message field contains either Log statistics; or psdo_wl_sync, correct? Because I am seeing all s_src messages making it through. The internal stats at level 1 show the filter some matching and some not. According to the stats, I think the filter is working, but I am seeing all of the s_src logs on the destination. Am I doing something wrong here? I am running OSE 3.25.
source s_src {
       system();
       internal();
};

filter f_statistics {
       message("Log statistics;") or message("psdo_wl_sync");
};

destination d_cribl { udp("0.0.0.0" port(1515) spoof_source(yes)); } ;

log { source(s_src); filter(f_statistics); destination(d_cribl); };
Artem Panasenkov
@artem.panasenkov_gitlab
try match("Log statistics;" value("MESSAGE") instead of message("Log statistics;")
clay584
@clay584
Trying now...
also, from experience, make sure if you spoof source, that any network gear (ie firewalls) wont deny and log that right back, causing a recursive loop and DOS lol
clay584
@clay584
image.png
Still the same result.
seeing these and other logs keep making it through
Yea, there are no firewalls between our syslog-ng collectors and the downstream workers.
Artem Panasenkov
@artem.panasenkov_gitlab
i missed a paren above, match("Log statistics;" value("MESSAGE"))
hmm, not sure off hand otherwise, consulting the doc's it should work. any errors logged on config reload?
clay584
@clay584
This is weird. I turned on syslog-ng -Fevdt to get tracing, and it actually shows a message being dropped, but I see it in the downstream system. WTF
[2022-04-06T20:57:24.496793] <<<<<< filter rule evaluation result; result='UNMATCHED - Dropping message from LogPipe', rule='f_statistics', location='/etc/syslog-ng/syslog-ng.conf:33:22', msg='0x7fe2fc003930'
And to answer your prior question syntax check says its fine
clay584
@clay584
I think I may open an issue on github.
Fabien Wernli
@faxm0dem
hey, I have a weird problem with ewmm destination
I have thousands of hosts sending messages with this destination driver, and they work fine
however, one server seems to get its messages interpreted wrong
all messages end up being parsed as PROGRAM: @syslog-ng with the raw json in the MESSAGE macro
both senders have the same syslog-ng version
syslog-ng-3.28.1-1.el7.x86_64
Fabien Wernli
@faxm0dem
additionally, a small fraction of the messages end up correctly parsed
the said host has a pretty high traffic : 20Million+/day
and about 1 million make it through correctly parsed
Gábor Nagy
@gaborznagy
@clay584 dropping a message in one log path element (e.g. a filter, parser) doesn't mean it cannot be catched by another log path.
I see you've opened syslog-ng/syslog-ng#3981 on GitHub, I'll check it and post a comment about my findings!
My first sentence about other log paths was meant to be in general, and not regarding your config (I haven't checked that yet).
Balazs Scheidler
@bazsi
@arteta22000 your pastebin links don't work anymore and without the samples I can't really comment on your question.
Fabien Wernli
@faxm0dem
bazsi: I'll make a pcap file for you, thanks for trying to help !
Shikhar Vashistha
@shikharvashistha

Hi, community I'm Shikhar Vashistha pursuing undergraduation, the project related to log rotation caught my attention, I've hands on experience with C and C++ and would like contribute the concerned area. Looking forward to hear from you.

Best Regards
Shikhar Vashistha

Shikhar Vashistha
@shikharvashistha
As suggested in the mailing list, for getting familiarize with the code base. I've tried building syslog-ng locally and raised a pull #3986 for solving #3255 marked as good first issue, looking forward for a review and feedback if any.
1 reply
Laszlo Pal
@vladx71:matrix.org
[m]

Hi,

I'm trying to drop some messages using in-list filter based on the content of MESSAGE

filter f_drop_linux_junk{
in-list ("/etc/syslog-ng/conf.d/sap_msg_drop.lst", value("MESSAGE"));
};

cat sap_msg_drop.lst
dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00

log {
source(s_udp1601_legacy);
source(s_tcp1601_legacy);
filter(f_drop_linux_junk);
destination(d_tmp_onefile);
flags(final);
};

So, in-list filter cannot be used with MESSAGE field? I'm using the same method to drop/forward messages using PROGRAM or HOST fields

Gábor Nagy
@gaborznagy

Hi @vladx71:matrix.org ! I think you forgot the not keyword, to negate the bool expression of the filter.

filter f_drop_linux_junk{
  not in-list ("/etc/syslog-ng/conf.d/sap_msg_drop.lst", value("MESSAGE"));
};

I.e. if you would like to drop matching patterns, then the not keyword should be used.

4 replies
Laszlo Pal
@vladx71:matrix.org
[m]
I mean the last part is the first few lines of my lst
Peter Czanik
@czanik
The in-list() filter also works on MESSAGE. However, it only works on full matches, not on partial matches.
9 replies
tncyysl
@tncyysl
can i collect from mac os x unified log ?
i just found nxlog enterprise edition. are there otherway ?
László Várady
@MrAnno
We've recently added system() source support for macOS. It reads /var/log/system.log, which is not ideal as that file is just a partial and text representation of some "unified logs", but it's something.
Laszlo Pal
@vladx71:matrix.org
[m]
Hi, I have a quite lame question :) At one of my customer I have to use spoofing on UDP (long painful story) and also I have to use throttle to keep SIEM happy. So, throttle and spoof-source should work together or because of the udp destionation throttle cannot work?
Laszlo Pal
@vladx71:matrix.org
[m]
it seems if I use extremely low e.g. 1 throttle value, the queue is slowly filling up, but not like when I use tcp destination... so I think with udp dest. throttle + spoof is not a good idea...
Balazs Scheidler
@bazsi
the spoof-source path bypasses queueing entirely.
And throttle is built into the queue egress path.
So spoof-source and throttle are incompatible at the moment.
static void
afinet_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options)
{
#if SYSLOG_NG_ENABLE_SPOOF_SOURCE
  AFInetDestDriver *self = (AFInetDestDriver *) s;

  /* NOTE: This code should probably be moved to the LogProto layer so that
   * spoofed packets are also going through the LogWriter queue */

  if (_is_spoof_source_enabled(self) && _is_message_spoofable(msg) && log_writer_opened(self->super.writer))
    {
      if (afinet_dd_spoof_write_message(self, msg, path_options))
        return;
    }
#endif
  log_dest_driver_queue_method(s, msg, path_options);
}
Laszlo Pal
@vladx71:matrix.org
[m]
Thank you Bazsi. It is strange, when I use an extreme low throttle value (e.g. 1) some disk buffer is being used (or maybe memory buffer if it also shows in syslog-ng-ctl stats)