Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
nbsd
@nbsd
@Kokan Thank you very much. This solved my problem.
nbsd
@nbsd
@Kokan @MrAnno In my unit test I need to set message parse options programmatically. The latest change making the parse_options a local structure breaks my unit test. How can I set the parse options from my test now? My current setup function does this: init_template_tests(); parse_options.flags |= LP_STORE_RAW_MESSAGE;
László Várady
@MrAnno

@nbsd Sorry about that. You can initialize parse_options using init_parse_options_and_load_syslogformat() from https://github.com/syslog-ng/syslog-ng/blob/3d271976f470832e573528a9eb9c7ca643f9fa67/libtest/msg_parse_lib.h#L33.

This function also loads the syslogformat module (which contains the BSD and the IETF syslog message parsers). If you don't need it, you can use msg_format_options_defaults(), msg_format_options_init(), and msg_format_options_destroy() instead: https://github.com/syslog-ng/syslog-ng/blob/3d271976f470832e573528a9eb9c7ca643f9fa67/lib/msg-format.h#L90-L92

László Várady
@MrAnno
After the initialization, you can set LP_STORE_RAW_MESSAGE or anything you need.
László Várady
@MrAnno
@nbsd I just saw your comment above parse_options.flags |= LP_STORE_RAW_MESSAGE;. It should work perfectly, but if you don't want to mess with parser options, you can skip the whole parsing step by calling log_msg_new_empty() and setting fields manually, for example:
LogMessage *msg = log_msg_new_empty();

log_msg_set_value(msg, LM_V_MESSAGE, message, strlen(message)); // sets $MSG
log_msg_set_value(msg, LM_V_PROGRAM, program, strlen(program)); // sets $PROGRAM
log_msg_set_value(msg, LM_V_HOST, "testhost", -1); // sets $HOST
nbsd
@nbsd
@MrAnno Thanks for the hints. I fear that this will not work, as I call the perftest_template() function which internally calls create_sample_message(). This function applies default parsing options to the message and passes it to my secure logging template. This cannot work, as my template needs the LP_STORE_RAW_MESSAGE flag set, otherwise it receives an empty string.
@MrAnno This would probably mean that I no longer can use the functions from the testing library in libtest/cr_template.c
nbsd
@nbsd
@MrAnno My unit test completes successfully if I do not call any functions from libtest/cr_template.c. This is unfortunate, as I would need to double code just in order to perform my test which is not a good way of doing it. I really would like to rely on the existing test API as much as possible but in my case this seems not to be feasible.
Peter Czanik
@czanik
@nbsd: compiled my syslog-ng git snapshot package for openSUSE with your PR:
linux-yv1e:~ # syslog-ng -V
syslog-ng 3 (3.25.1.207.g3d27197.dirty)
Config version: 3.25
Installer-Version: 3.25.1.207.g3d27197.dirty
if all goes well, I'll do some minimal testing tomorrow
I also have man pages and utilities packaged:
   This manual page was written by the Airbus Secure Logging Team
   <secure-logging@airbus.com>.
nbsd
@nbsd
@czanik Thank you Peter :-)
@czanik @MrAnno @Kokan My PR now passes all Travis CI checks and builds correctly. However, the MacOS job still fails. How can I fix this? I am a Mac user so I could do some tests But I am not able to see any error messages that might give me a hint about what the problem on the Mac may be.
Peter Czanik
@czanik
@nbsd I found this:

BUILD SUCCESSFUL in 46s
16 actionable tasks: 16 executed
make[2]: Target all-am' not remade because of errors. make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 /Applications/Xcode_11.3.1.app/Contents/Developer/usr/bin/make --no-print-directory all-recursive gcc -DHAVE_CONFIG_H -I. -I./lib -I./modules -I./lib -I./modules -I./modules/cryptofuncs -I./modules/cryptofuncs -I/usr/local/Cellar/glib/2.62.4/include -I/usr/local/Cellar/glib/2.62.4/include/glib-2.0 -I/usr/local/Cellar/glib/2.62.4/lib/glib-2.0/include -I/usr/local/opt/gettext/include -I/usr/local/Cellar/pcre/8.43/include -I./lib/eventlog/src -I./lib/eventlog/src -I/usr/local/Cellar/pcre/8.43/include -I/usr/local/Cellar/openssl/1.0.2t/include -DHAVE_SOCKADDR_SA_LEN -DLIBNET_BSDISH_OS -DLIBNET_BSD_BYTE_SWAP -D_DEFAULT_SOURCE -I/usr/local/Cellar/libdbi/0.9.0/include/dbi -I/usr/local/Cellar/libdbi/0.9.0/include/dbi/dbi -I/usr/local/Cellar/ivykis/0.42.4/include -D_GNU_SOURCE -D_DEFAULT_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Wshadow -Wno-stack-protector -Wno-unused-parameter -Wno-variadic-macros -O2 -g -pthread -Wno-initializer-overrides -MT modules/cryptofuncs/slogkey/slogkey-slogkey.o -MD -MP -MF modules/cryptofuncs/slogkey/.deps/slogkey-slogkey.Tpo -c -o modules/cryptofuncs/slogkey/slogkey-slogkey.otest -f 'modules/cryptofuncs/slogkey/slogkey.c' || echo './'`modules/cryptofuncs/slogkey/slogkey.c
modules/cryptofuncs/slogkey/slogkey.c:28:10: fatal error: 'endian.h' file not found

include <endian.h>

     ^~~~~~~~~~

1 error generated.
make[2]: [modules/cryptofuncs/slogkey/slogkey-slogkey.o] Error 1
make[1]:
[all-recursive] Error 1
make: * [all] Error 2

nbsd
@nbsd
@czanik This is strange. endian.h does exist on my system running MacOS 10.14.6
László Várady
@MrAnno
Hm, weird. On macOS, it might be under /usr/include/machine/ instead of /usr/include.
nbsd
@nbsd
@Kokan Building on MacOS with the compatibility layer for hsearch_r produces the following error ld: library not found for -lcrypto.35
@Kokan This library is present on my system
Kókai Péter
@Kokan
@nbsd I don't have a macos to check out, but it seems you solved the issue (macos build passed on latest commit)
do you still have this issue locally ?
also I see macos is fine now but travis is broken :(
there is an issue with copyright, but I would say do that at last and first focus on the build
Peter Czanik
@czanik
@nbsd : in the end I did not have time for testing, but now also my fedora/rhel git snapshot packages are built with your PR
nbsd
@nbsd
@czanik Interesting. Thanks for informing. What RPM SPEC file did you use to create the RPM?
nbsd
@nbsd
@Kokan @MrAnno @Kokan Thanks for your excellent support for creating a valid PR. All checks do now pass. Therefore, I would be happy if the core team would review the PR and provide me with feedback.
it's about 99% the same as bundled in syslog-ng sources at https://github.com/syslog-ng/syslog-ng/tree/master/packaging/rhel
nbsd
@nbsd
@czanik I have now created an independent module for the secure logging functionality. Can you try to build your RPM using the new layout and provide me with feedback on this?
Peter Czanik
@czanik
@nbsd I already updated to git snapshot packages the latest modifications of your PR :)
So, anybody who installed it since yesterday has secure logging as module
linux-yv1e:~ # syslog-ng -V
syslog-ng 3 (3.25.1.239.g0535e8a)
Config version: 3.25
Installer-Version: 3.25.1.239.g0535e8a
Revision:
Module-Directory: /usr/lib64/syslog-ng
Module-Path: /usr/lib64/syslog-ng
Include-Path: /usr/share/syslog-ng/include
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,dbparser,disk-buffer,examples,graphite,hook-commands,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,sdjournal,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,xml,kafka,http,timestamp,azure-auth-header,secure-logging
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
nbsd
@nbsd
@czanik Thanks very much for the update :-D
jhgc165
@jhgc165

Hey All, I'm new to syslog-ng and interested in developing a custom Java Destination. I have read the "Getting-started.pdf". The document says I must extend either the TextLogDestination class or the StructuredLogDestination class. There are no comments in the code of these classes, so I have a few questions:

What is the purpose of the TextLogDestination vs the StructuredLogDestination class? How do I know which one I should extend.

Additionally, as there are no comments in the code, I do not know the purpose of the abstract methods, and therefore do not know how I should properly implement the methods for their intended use. For example, what is the purpose of open(), close(), isOpened(), getNameUniqOptions() etc.

Budai Laszlo
@lbudai
@jhgc165 : the main difference between Text and Structured is the send() method: Text receives the log message as a string while Structured is a LogMessage object. Text is simpler, so if you don’t need fields from LogMessage object then use Text
@jhgc165 : destinations are sending messages to their final place. this is a resource (file, socket, database, whatever) that need to be opened/closed in most of the cases. If you want to send logs to a destination with the help of a 3rd party library that does not support open/close and you have to reconnect all the time, just return true from open, and false from close.
jhgc165
@jhgc165
Thanks for the assistance. @lbudai do you know anything about getNameUniqOptions()?
derik709
@derik709
New to this group older syslog-ng user. Looking to migrate from rsyslog to syslog-n, in very large org. I wanted to ask this group as I am sure I will find info in goggle but wanted to make sure I find the best guide most relevant. Main reason for shifting is to focus on removing certain PHI data from application output vs forcing to encrypt. Thanks in advance to any info you can provide..
Budai Laszlo
@lbudai
@jhgc165 syslog-ng is able to store LogPipe information(position, diskq file, ...) across reloads and even restarts (in the persist file) and can provide statistic counters (number of processed, dropped messages, etc...).
These data are indetified by persist name and this name should be unique.
What makes a LogPipe element (source, destination, ...) unique? Options (port, host, filename).
In case of a java destination users can define a unique name for identifying the destination by implementing this method.
This name will be then used in statistics and for example identifying diskq files.
Budai Laszlo
@lbudai
@derik709 so you want to remove sensitive information from your log messages? Rewrite rules?
derik709
@derik709
yes, trying to replace our rsyslog..using rhel 6...found that 3.13 is latest for that release..but yes take sensitive data and send to dev/null..Just looking for as much 411 as i can find..
jhgc165
@jhgc165
@lbudai Awesome, thank you very much for your help.
nbsd
@nbsd
@lbudai @Kokan @MrAnno The secure logging functionality has now been moved into a dedicated module which passes all checks of the build environment. I'd like to request a review of this module by the core team.
Budai Laszlo
@lbudai
@nbsd : thanks! during next week we will check it
nbsd
@nbsd
@lbudai Great :-) I'm looking forward to your comments. May thanks to the core team for their superb support in getting the PR into the right form.