syslog/rsyslog
Contents
About
- a Rocket-fast SYStem for LOG processing
- Reliable system and kernel logging daemon
- Development started in 2004 by
Rsyslog is a multi-threaded implementation of syslogd (a system utility providing support for message logging), with features that include:
- reliable syslog over TCP, SSL/TLS and RELP
- on-demand disk buffering
- email alerting
- writing to MySQL or PostgreSQL databases (via separate output plugins)
- permitted sender lists
- filtering on any part of the syslog message
- on-the-wire message compression
- fine-grained output format control
- failover to backup destinations
- enterprise-class encrypted syslog relaying
It is the default syslogd on current Debian systems. Rsyslog is the prevailingly installed syslog server in Linux distributions and thus worth a look. In upcoming releases of the distributions systemd-journald will take over this role.
Licenses:
- LGPL-3.0
- GPL-3.0
- Apache-2.0
Links
github.com rsyslog/rsyslog-docker - example appliance config
access.redhat.com/documentation - Configuring rsyslog on a Logging Server
DAVID LANG'S ;LOGIN: SERIES ON LOGGING TOPICS
- Oldie but goldie (2013)
- The articles are attached as files - just in case:
- Reliable Event Logging Protocol (RELP)
Versioning
- Version naming scheme changed
- Version 8.40.0 [v8-stable] 2018-12-11
- Version 8.1901.0 (aka 2019.01) 2019-01-22
Installation
Install rsyslog Debian
1 apt install rsyslog
Install rsyslog RedHat
1 dnf install rsyslog
Configure
Debian default config
The default config in Debian GNU/Linux
/etc/rsyslog.conf
Example configs
Some example configs that are included by the main config file ordered by numerical precedence. They have been hidden as comments. Their visibility may be toggled with the eye in the top bar.
/etc/rsyslog.d/20-ufw.conf
/etc/rsyslog.d/45-frr.conf
/etc/rsyslog.d/postfix.conf
Redhat default config
The default config in Redhat
/etc/rsyslog.conf
Requirements
- rsyslog daemons communicate securely
- hardened crypto
- valid certificates
- Buffering of messages on clients
- messages (at least 30 days) are stored on the client during loss of connection to upstream sink
- maximum log-level: info
- redundancy of syslog-sinks
- cluster-ip
Design decisions
Coexistence of rsyslog with systemd-journald
Rsyslog is in the process of being substituted in the distributions by systemd-journald. By default now the system-log-socket /dev/log has been granted to systemd-journald. Systemd-journald passes the log-info over to other daemons via the syslog-socket
/run/systemd/journal/syslog
Please see:
freedesktop.org Wiki - Writing syslog Daemons Which Cooperate Nicely With systemd
systemd.io - Writing syslog Daemons Which Cooperate Nicely With systemd
For this reason the SystemD socket syslog.socket has to started. As documented in
/lib/systemd/system/syslog.socket
in the rsyslog.service some statements have to be added in the sections
[Unit] - Requires=syslog.socket
syslog.socket is started before the syslog.service
[Install] - Alias=syslog.service
syslog.service is activated by syslog.socket (socket activation)
This should be done via a systemd service override e. g. in
/etc/systemd/system/rsyslog.service.d/override.conf
Creating an override may be accomplished by
1 systemctl edit rsyslog.service
Ansible and systemd units of type "static"
Systemd units of type "static"
don't have an own section [Install] and SystemD therefor does not know how to enable/disable/mask/… this unit.
may be referenced as dependencies e. g. by Requires=…
Deactivation of static units is not supported in Ansible, but is reported as Changed: True (always), which is kind of misleading. https://github.com/ansible/ansible/issues/46247
For example the following units cannot be disabled by Ansible:
systemd-journald.socket
systemd-journald-dev-log.socket
systemd-journald.service
To follow the recommendation of the rsyslog project systemd-overrides have been used, to implement a solution in which systemd-journald passes the sysog-data via a unix-domain-socket over other daemons.
A systemd-journald override has been created to enable such socket.
/etc/systemd/journald.conf.d/override.conf
Reliable Event Logging Protocol (RELP)
The transmission protocol RELP has been chosen to minimize message loss.
RELP is inspired by
Description of the protocol
- Provides the ability to receive syslog messages via the reliable RELP protocol. This module requires librelp to be present on the system. From the user’s point of view, imrelp works much like imtcp or imgssapi, except that no message loss can occur. Please note that with the currently supported relp protocol version, a minor message duplication may occur if a network connection between the relp client and relp server breaks after the client could successfully send some messages but the server could not acknowledge them. The window of opportunity is very slim, but in theory this is possible. Future versions of RELP will prevent this. Please also note that rsyslogd may lose a few messages if rsyslog is shutdown while a network connection to the server is broken and could not yet be recovered. Future version of RELP support in rsyslog will prevent that. Please note that both scenarios also exists with plain tcp syslog. RELP, even with the small nits outlined above, is a much more reliable solution than plain tcp syslog and so it is highly suggested to use RELP instead of plain tcp. Clients send messages to the RELP server via omrelp.
Documentation
Secure cryptography
- During the transmission of sensitive log data encryption is generally to be applied.
- Cryptography is configured to use OpenSSL instead of GNUTLS, because the ladder project seems to be maintained more actively and therefor is more appropriate to the security requirements.
- Insecure cipher-protocols have been deactivated to comply.
- Servers and client identify themselves mutually based on the DNS names in their certificates.
The configured cryptography works and validates successfully (even against the system CA-bundle).
Queues
All network-facing output modules have been aided by a disk-assisteed (hybrid) queue of type LinkedList, which size is limited to 1GiB. Please see:
The new configuration
The main configuration file has been freed from the module configurations, which have been moved to included files. What remains is the body which contains the definition of the log files.
- The handling of the standard log files should not change so that the administrators continue to feel “familiar” with the systems.
- The standard log files should also not be flooded.
- Modules can only be loaded once and it must be possible to specify options at module level when loading them. In order to maintain full control over the configuration, the module configurations have been removed.
/etc/rsyslog.conf
Included files - /etc/rsyslog.d
Filesystem structure of the configuration
Client
1 -rw-r--r--. 1 root root 3313 19. Aug 21:32 rsyslog.conf
2 -rw-r--r--. 1 root root 3227 13. Feb 2024 rsyslog.conf.dist
3
4 /etc/rsyslog.d:
5 insgesamt 16
6 -rw-r--r--. 1 root root 488 20. Aug 11:46 11_global.conf
7 -rw-r--r--. 1 root root 2519 20. Aug 12:41 21_modules_client.conf
8 -rw-r--r--. 1 root root 2322 20. Aug 11:46 22_modules_server.conf
9 -rw-r--r--. 1 root root 2607 20. Aug 11:46 31_inputs.conf
Server
1 -rw-r--r--. 1 root root 3313 20. Feb 15:58 /etc/rsyslog.conf
2 -rw-r--r--. 1 root root 3227 13. Feb 2024 /etc/rsyslog.conf.dist
3
4 /etc/rsyslog.d:
5 insgesamt 20
6 -rw-r--r--. 1 root root 488 20. Aug 11:46 11_global.conf
7 -rw-r--r--. 1 root root 2519 20. Aug 12:41 21_modules_client.conf
8 -rw-r--r--. 1 root root 923 20. Aug 13:13 81_to_graylog.conf
9 -rw-r--r--. 1 root root 1285 20. Aug 13:08 82_audit_to_network.conf
10 -rw-r--r--. 1 root root 1173 20. Aug 13:08 83_systemd_to_network.conf
11_global.conf
The configuration of the NetstreamDriver can only be done in global scope until Version 8.2108.0 (RedHat EL 8 - 8.2102.0-15.el8 (aka 2021.02)).
/etc/rsyslog.d/11_global.conf
1 ### GLOBAL
2
3 # For more information please see:
4 # https://www.rsyslog.com/doc/rainerscript/global.html
5
6 # Configure defaults of Netstream drivers
7 global(
8 DefaultNetstreamDriver="ossl"
9 DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-chain.pem"
10 DefaultNetstreamDriverKeyFile="/etc/pki/tls/private/rocky8-client1.example.com.pem"
11 DefaultNetstreamDriverCertFile="/etc/pki/tls/certs/rocky8-client1.example.com.crt")
21_modules_client.conf
Preset modules have been commented out in the main configuration file and are loaded in this file instead.
/etc/rsyslog.d/21_modules_client.conf
1 ### CLIENT MODULES
2
3 # For more information please see:
4 # https://www.rsyslog.com/doc/v8-stable/configuration/modules/index.html
5
6 # This module MUST be loaded right at the top of rsyslog.conf, otherwise stats
7 # may not get turned on in all places.
8 module(load="impstats"
9 # Sets the interval, in seconds at which messages are generated. Please
10 # note that the actual interval may be a bit longer. We do not try to be
11 # precise and so the interval is actually a sleep period which is
12 # entered after generating all messages. So the actual interval is what
13 # is configured here plus the actual time required to generate messages.
14 # In general, the difference should not really matter. values: Integer,
15 # default: 300
16 interval="300")
17 # provides access to the systemd journal
18 #module(load="imjournal"
19 #config.enabled="off"
20 # File to store the position in the journal
21 #StateFile="imjournal.state")
22 # reads kernel messages (the same are read from journald)
23 #module(load="imklog"
24 #config.enabled="off")
25 # provides --MARK-- message capability
26 #module(load="immark"
27 #config.enabled="off")
28 # provides support for local system logging (e.g. via logger command)
29 module(load="imuxsock"
30 # Turn off message reception via local log socket; local messages are
31 # retrieved through imjournal now.
32 SysSock.Use="on")
33 # The omfile plug-in provides the core functionality of writing messages to files
34 # residing inside the local file system. Both files named with static names as
35 # well as files with names based on message content are supported by this module.
36 module(load="builtin:omfile"
37 # Set the default template to be used if an action is not configured to
38 # use a specific template. -> Use default timestamp format.
39 Template="RSYSLOG_TraditionalFileFormat")
40 # The omfwd plug-in provides the core functionality of traditional message
41 # forwarding via UDP and plain TCP. It is a built-in module that does not need to
42 # be loaded. Please see:
43 # https://www.rsyslog.com/doc/configuration/modules/omfwd.html
44 module(load="builtin:omfwd")
45 # This module supports sending syslog messages over the reliable RELP protocol.
46 # Please see:
47 # https://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
48 module(load="omrelp"
49 # Permits to specify the TLS library used by librelp. values: "openssl",
50 # "gnutls" (default).
51 tls.tlslib="openssl")
22_modules_server.conf
- Only on the syslog server
- The imrelp and imtcp modules are loaded on the server with module options for configuring the TLS stack.
/etc/rsyslog.d/22_modules_server.conf
1 #
2 # Ansible managed
3 #
4
5 ### SERVERS MODULES
6
7 # For more information please see:
8 # https://www.rsyslog.com/doc/v8-stable/configuration/modules/index.html
9
10 # Provides the ability to receive syslog messages via the reliable RELP protocol
11 # Please see:
12 # https://www.rsyslog.com/doc/v8-stable/configuration/modules/imrelp.html
13 module(load="imrelp"
14 # Permits to specify the TLS library used by librelp. values: "openssl",
15 # "gnutls" (default).
16 tls.tlslib="openssl")
17 # Provides TCP syslog reception for parameters see
18 # http://www.rsyslog.com/doc/imtcp.html
19 module(load="imtcp"
20 # Selects network stream driver for all inputs using this module.
21 # values: none (default), "ptcp", "gtls", "ossl".
22 StreamDriver.Name="ossl"
23 # Sets the driver mode for the currently selected network stream driver.
24 # <number> is driver specific. Network stream driver implement a TLS
25 # protected transport via the GnuTLS or OpenSSL library. values:
26 # 0 - unencrypted transmission (just like ptcp driver) (default)
27 # 1 - TLS-protected operation
28 StreamDriver.Mode="1"
29 # Sets stream driver authentication mode. Possible values and meaning
30 # depend on the network stream driver used. values: none (default),
31 # "anon", "x509/fingerprint",
32 # "x509/certvalid", "x509/name"
33 StreamDriver.AuthMode="x509/name"
34 # The “gnutls priority string” parameter in rsyslog offers enhanced
35 # customization for secure communications, allowing detailed
36 # configuration of TLS driver properties. This includes specifying
37 # handshake algorithms and other settings for GnuTLS, as well as
38 # implementing OpenSSL configuration commands. Initially developed for
39 # GnuTLS, the “gnutls priority string” has evolved since version
40 # v8.1905.0 to also support OpenSSL, broadening its application and
41 # utility in network security configurations. This update signifies a
42 # key advancement in rsyslog’s capabilities, making the “gnutls priority
43 # string” an essential feature for advanced TLS configuration. values:
44 # none (default), string
45 gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.1
46 MinProtocol=TLSv1.2")
31_inputs.conf
- Only on the syslog server
- This configuration shows the syslog concentrator in particular the configuration of the input modules for TCP (tcp/6514) and RELP (tcp/2514) for maximum security.
/etc/rsyslog.d/31_inputs.conf
1 #
2 # Ansible managed
3 #
4
5 ### INPUTS
6
7 # For more information please see:
8 # https://www.rsyslog.com/doc/v8-stable/configuration/modules/index.html
9
10 # Provides the ability to receive syslog messages via the reliable RELP protocol
11 # Please see:
12 # https://www.rsyslog.com/doc/v8-stable/configuration/modules/imrelp.html
13 input(type="imrelp"
14 # Starts a RELP server on selected port Default: None, mandatory
15 port="2514"
16 # If set to “on”, the RELP connection will be encrypted by TLS, so that
17 # the data is protected against observers. Please note that both the
18 # client and the server must have set TLS to either “on” or “off”.
19 # Default: off
20 tls="on"
21 # Sets the mode used for mutual authentication. Supported values are
22 # either “fingerprint“ or “name”. Default: none
23 tls.authmode="name"
24 # Peer Places access restrictions on this forwarder. Only peers which
25 # have been listed in this parameter may be connected to. This guards
26 # against rouge servers and man-in-the-middle attacks. The validation
27 # bases on the certificate the remote peer presents.
28 tls.permittedpeer="*.example.com"
29 # The CA certificate that is being used to verify the client
30 # certificates. Has to be configured if "TLS.AuthMode" is set to
31 # "fingerprint" or "name"
32 tls.cacert="/etc/pki/ca-trust/source/anchors/ca-chain.pem"
33 # The machine certificate that is being used for TLS communication.
34 tls.mycert="/etc/pki/tls/certs/rocky8-server1.example.com.crt"
35 # The machine private key for the configured "TLS.MyCert".
36 tls.myprivkey="/etc/pki/tls/private/rocky8-server1.example.com.pem"
37 # The setting can be used if tls.tlslib is set to “openssl” to pass
38 # configuration commands to the openssl libray. OpenSSL Version 1.0.2 or
39 # higher is required for this feature. A list of possible commands and
40 # their valid values can be found in the documentation:
41 # https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html The
42 # setting can be single or multiline, each configuration command is
43 # separated by linefeed (n). Command and value are separated by equal
44 # sign (=).
45 tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.1
46 MinProtocol=TLSv1.2")
47 # Provides TCP syslog reception for parameters see
48 # http://www.rsyslog.com/doc/imtcp.html
49 input(type="imtcp"
50 # Starts a TCP server on selected port. Default: None, mandatory
51 port="6514")
81_to_graylog.conf
* Only on the syslog client * All events are sent to the graylog
/etc/rsyslog.d/81_to_graylog.conf
1 #
2 # Ansible managed
3 #
4 # Order: 81
5 # Name: To Graylog
6 # Description: Send all events to Graylog
7 #
8
9 if ($syslogseverity <= 6)
10 then {
11 action(Type="omfwd"
12 Name="omfwd-tls-rocky8-server.example.com:6514"
13 Target="rocky8-server.example.com"
14 Port="6514"
15 Protocol="tcp"
16 StreamDriver="ossl"
17 StreamDriverMode="1"
18 StreamDriverAuthMode="x509/name"
19 StreamDriverPermittedPeers="*.example.com"
20 gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.1
21 MinProtocol=TLSv1.2"
22 queue.type="LinkedList"
23 queue.size="200000"
24 # unique name prefix for spool files
25 queue.filename="all_to_graylog.queue"
26 queue.workerThreads="2"
27 queue.maxFileSize="1g"
28 # 1gb space limit (use as much as possible)
29 queue.maxdiskspace="1g"
30 # save messages to disk on shutdown
31 queue.saveonshutdown="on"
32 # infinite retries if host is down
33 action.resumeRetryCount="-1"
34 )
35 }
82_audit_to_network.conf
- Only on the syslog client
Events with the priority INFO or lower (<=6) of the facility local6, in which the Auditd event dispatcher Daemon audisp is delivered, are sent directly to the syslog server and processed further.
/etc/rsyslog.d/82_audit_to_network.conf
1 #
2 # Ansible managed
3 #
4 # Order: 82
5 # Name: Audit to network
6 # Description: Send Auditd events to syslog aggregator and stop processing to
7 # avoid filling up local logs since auditd already wrote a copy.
8 #
9
10 if ($syslogseverity <= 6 and
11 $syslogfacility-text == "local6")
12 then {
13 action(Type="omrelp"
14 Name="omrelp-tls-rocky8-server.example.com:2514-auditd"
15 Target="rocky8-server.example.com"
16 Port="2514"
17 TLS = "on"
18 TLS.CaCert="/etc/pki/ca-trust/source/anchors/ca-chain.pem"
19 TLS.MyCert="/etc/pki/tls/certs/rocky8-client1.example.com.crt"
20 TLS.MyPrivKey="/etc/pki/tls/private/rocky8-client1.example.com.pem"
21 TLS.TlsCfgCmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.1
22 MinProtocol=TLSv1.2"
23 TLS.AuthMode="name"
24 TLS.PermittedPeer="*.example.com"
25 #TLS.PermittedPeer=["syslogA","syslogB"]
26 queue.type="LinkedList"
27 queue.size="200000"
28 # unique name prefix for spool files
29 queue.filename="audit_to_network.queue"
30 queue.workerThreads="2"
31 queue.maxFileSize="1g"
32 # 1gb space limit (use as much as possible)
33 queue.maxdiskspace="1g"
34 # save messages to disk on shutdown
35 queue.saveonshutdown="on"
36 # infinite retries if host is down
37 action.resumeRetryCount="-1"
38 )
39 ### DON'T FILL UP LOCAL LOGS
40 & stop
41 }
83_systemd_to_network.conf
- Only on the syslog client
- Systemd messages reveal whether services have been started or stopped. Messages about the start and stop of Rsyslogd or Auditd are also logged here again. This can also be determined using the data collected by Auditd, and deactivating this forwarding is very easy to achive by deleting the file.
/etc/rsyslog.d/83_systemd_to_network.conf
1 #
2 # Ansible managed
3 #
4 # Order: 83
5 # Name: Systemd to network
6 # Description: Log if a program state has changed to support detecting (D)DoS
7 # attacks.
8 #
9
10 if ( $programname == [
11 "systemd"
12 ])
13 then {
14 action(Type="omrelp"
15 Name="omrelp-tls-rocky8-server.example.com:2514-systemd"
16 Target="rocky8-server.example.com"
17 Port="2514"
18 TLS = "on"
19 TLS.CaCert="/etc/pki/ca-trust/source/anchors/ca-chain.pem"
20 TLS.MyCert="/etc/pki/tls/certs/rocky8-client1.example.com.crt"
21 TLS.MyPrivKey="/etc/pki/tls/private/rocky8-client1.example.com.pem"
22 TLS.TlsCfgCmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.1
23 MinProtocol=TLSv1.2"
24 TLS.AuthMode="name"
25 TLS.PermittedPeer="*.example.com"
26 #TLS.PermittedPeer=["syslogA","syslogB"]
27 queue.type="LinkedList"
28 queue.size="200000"
29 # unique name prefix for spool files
30 queue.filename="systemd_to_network.queue"
31 queue.workerThreads="2"
32 queue.maxFileSize="1g"
33 # 1gb space limit (use as much as possible)
34 queue.maxdiskspace="1g"
35 # save messages to disk on shutdown
36 queue.saveonshutdown="on"
37 # infinite retries if host is down
38 action.resumeRetryCount="-1"
39 )
40 }
== Recommendations of the rsyslog project
Queues
- It may be necessary to adapt the queues on the servers to the workload.
- Especially in the aggregators, switch to hybrid (memory and disk)
- By default
- only 1/16MiB in the action queues
- only in-memory queues in “direct” mode, which are not really queues
- offers potential for message loss
- Recommendations of the Rsyslog project
Type: In general, it is advised to use LinkedList mode if in doubt
- Disk-assisted: it is recommended to use a disk-assisted linked list in-memory queue in front of each database and “send via tcp” action
- Disk Queues: For regular use cases, this queue mode is not recommended.
QueueMaxDiskSpace: In general, it is a good idea to limit the physical disk space even if you dedicate a whole disk to rsyslog.
fullDelaymark, lightDelayMark, timeoutEnqueue: UDP cannot be throttled
- Sources
Rsyslog security
1 systemd-analyze security rsyslog.service
Trouble shooting
Trouble shooting options
Please see
Perform a config check
Dump full config (combine with the config check to prevent errors).
1 rsyslogd -N 2 -o /tmp/rsyslog_full.conf
Trouble shooting module loads
- Never load a module twice. Rsyslog seems not to be recognizing valid module parameters again.
RELP crypto
/var/log/messages
1 Sep 5 16:08:44 hostname rsyslogd: omrelp[hostname.domain.tld:2514]: authentication error 'certificate validation failed', peer is '' - DISABLING action [v8.2106.0 try https://www.rsyslog.com/e/2353 ]
2 Sep 5 16:08:44 hostname rsyslogd: omrelp[hostname.domain.tld:2514]: error 'TLS handshake failed [gnutls error -43: Error in the certificate.]', object 'conn to srvr hostname.domain.tld:2514' - action may not work as intended [v8.2106.0 try https://www.rsyslog.com/e/2353 ]
Probably the TLS.PriorityString is configured to "modern", no communication is established and thus no peer can be identified. You may try
1 TLS.PriorityString="SECURE256:SECURE192:SECURE128:-VERS-TLS1.0:-VERS-TLS1.1"