rsyslog
rsyslog is the system logging daemon used by many Linux distributions. It receives log messages from the kernel, services and the network, and routes them to files, databases or other servers according to rules. Matching works on facility and priority or on message content, which makes rsyslog the usual tool for sending a copy of server logs off the machine.
How rsyslog processes a message
rsyslog works as a three-stage pipeline, described in its basic structure documentation. Inputs receive messages, for example imuxsock for the local syslog socket and imtcp for the network. Each message enters a ruleset, where rules are evaluated from top to bottom, and every rule pairs a filter with actions such as writing a file or forwarding to a server. The main file is /etc/rsyslog.conf, which usually includes snippets from /etc/rsyslog.d/. The project recommends the RainerScript syntax for new configuration and treats the older sysklogd and dollar-directive styles as deprecated.
The classic filter is a selector such as authpriv.*. The facility names the subsystem that logged, for example auth, authpriv, cron, daemon, kern, mail and local0 to local7. The priority runs from debug to emerg. A priority in a selector matches that level and everything more severe, and = matches only that level. The numbering follows RFC 5424. Templates control how a message is formatted for an output.
Forwarding logs to a central server
Forwarding is done by the built-in omfwd output, which sends over UDP, TCP or TLS. Its documentation advises TCP with TLS over plain TCP or UDP, and a queue so that an unreachable server does not block local logging. A sender sketch:
global(workDirectory="/var/spool/rsyslog")
action(type="omfwd" target="logs.example.com" port="6514" protocol="tcp"
StreamDriver="ossl" StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="logs.example.com"
StreamDriver.CAFile="/etc/ssl/rsyslog/ca.pem"
StreamDriver.CertFile="/etc/ssl/rsyslog/client.pem"
StreamDriver.KeyFile="/etc/ssl/rsyslog/client.key"
action.resumeRetryCount="-1" queue.type="LinkedList"
queue.filename="fwd_logs" queue.saveOnShutdown="on")Port 6514 is the port that RFC 5425 allocates for syslog over TLS. The receiver loads imtcp with the same stream driver settings and a PermittedPeer list naming the clients it accepts. Check the file with rsyslogd -N1 before you restart the service.
The queue lines matter. action.resumeRetryCount="-1" retries without limit, and queue.filename lets unsent messages spill to disk when memory fills, so a short outage on the log host should not lose messages. Test it by stopping the receiver for a while and starting it again.
Reliability, trust and common mistakes
Plain syslog was not built for hostile networks. NIST SP 800-92 notes that UDP gives no assurance that entries arrive, that most implementations do no access control so any host can send to the server, and that attackers can flood it with bogus data. So restrict who can reach the receiving port with a firewall, and use TLS with PermittedPeer so only your own servers are heard.
TCP is better than UDP but still not lossless. For acknowledged delivery rsyslog offers RELP through omrelp and imrelp. The imrelp documentation says it is much more reliable than plain TCP, while admitting a rare duplicate or a loss at shutdown. Disk queues help with outages, but queue.saveOnShutdown covers only an orderly shutdown, not a crash or power loss.
Two smaller traps: the short legacy forwarding forms that start with one or two at-signs bypass queues, so write the explicit action() form instead, and file names built from sender-supplied values such as the hostname need the secpath-replace option so a sender cannot pick the path.
Why off-server logs matter after a compromise
A local log is only as trustworthy as the root account of the machine that holds it. NIST SP 800-92 points out that many rootkits are designed to alter logs and remove traces of their own installation. Anyone who gains root can edit or delete files under /var/log, or stop the daemon. A copy that already sits on another host preserves what happened up to that moment, and a server that suddenly goes silent in the central log is a signal in itself.
Keep the log host separate from the servers it records, with its own credentials and access limited to the few people who investigate. Our monitoring guide explains what to look for once the logs are in one place, and the small server SIEM guide shows the next step. During an incident the central copy is the evidence you can still trust, so use it in the breach response timeline. The local journal is read with journalctl, per-command and per-file records come from auditd, and a SIEM correlates the combined stream.
Related terms: journalctl, auditd, SIEM, Incident response, File integrity monitoring.
Frequently asked questions
What is rsyslog used for?
rsyslog receives log messages from the kernel, services and other hosts, and routes them to files, databases or remote servers by rules. On servers it is most often used to write system logs and to forward a copy to a central log host.
What are syslog facilities and priorities?
The facility names the subsystem that produced a message, such as auth, cron, kern or local0 to local7. The priority is its severity, from debug to emerg, and a selector like authpriv.* matches the chosen priority and everything more severe.
How do I forward logs to a remote server with rsyslog?
Add an action() with type="omfwd", the target host, a port and protocol="tcp", and preferably TLS settings and a queue. Restart rsyslog after checking the file with rsyslogd -N1.
Should I use UDP, TCP, TLS or RELP for forwarding?
The rsyslog documentation recommends TCP with TLS over plain TCP or UDP, and RELP when acknowledged delivery is critical. UDP gives no assurance that messages arrive.
Is rsyslog still needed if I have the systemd journal?
The journal covers local collection, but rsyslog is a common choice for rule-based routing and for forwarding logs to another host. Many setups use both, with rsyslog receiving messages from the journal.
Want to see where your own server stands?
Run the free, read-only server check, or open the Security Lab and watch the detect, contain, recover, verify loop in action.
Get your free server checkOpen the Security Lab