The Linux system log files are invaluable when it comes to troubleshooting. Almost always located in /var/log, the log files are typically named after the processes that log to them, such as:
/var/log/kern.log/var/log/cron.log/var/log/mail.log
…and so on. When you know you’re investigating a problem with cron, the /var/log/cron logfile is an obvious starting point.
Catch-all
There are some logfiles that log event from multiple processes, such as /var/log/messages and /var/log/debug, but they will have restrictions on what they log, either in terms of the severity of the error message or the process (“facility”) that logs it.
These logfiles are useful, but sometimes in the process of investigating a problem, you want to see exactly what was logged at around 14:15 yesterday. That means looking through multiple log files and, if it seems that entries in multiple log files may be related to the issue you’re investigating, you need to look carefully to establish the order in which events occurred.
Everything
For many years now, we’ve implemented a /var/log/everything log file on the servers we support. As the name implies, pretty much everything is logged there in addition to the usual log files.
The contemporary system logging package is typically rsyslog, but some older systems use syslog‑ng. Here’s how we set it up on each of those:
syslog-ng
The configuration file is typically named syslog‑ng.conf, and contains a number of sections. Somewhere near the top the source of logging information will be defined, such as:
source s_src { unix-dgram("/dev/log"); internal();
file("/proc/kmsg" program_override("kernel"));
};
Later in the file “destinations” will be defined. We add one for the new log file:
destination d_all { file("/var/log/everything"); };
Finally, later still will be “log” lines, where we map the source to the destination. Some of the existing log statements may have flags(final) in them, which means that no more logging for that particular source and filter will take place. Be sure to put the following line before any flags(final) lines, perhaps as the first log line:
log { source(s_src); destination(d_all);};
rsyslog
The configuration file is typically /etc/rsyslog.conf, but often it includes a line similar to:
$IncludeConfig /etc/rsyslog.d/*.conf
That allows us to put additional configuration files in /etc/rsyslog.d where rsyslog will read them. We create, imaginatively, /etc/rsyslog.d/tiger.conf which contains in part:
# Copy all log messages to /var/log/everything *.* -/var/log/everything
The ‘-‘ before the log filename tells rsyslog not to call fsync() after writes, which speeds things up.
Rotation
A /var/log/everything file set up as described above will rapidly grow, so it is important to rotate it frequently. Log rotation is a way of closing off the existing log file, starting a new one, optionally compressing the old one, and removing very old files.
Create a file in /etc/logrotate.d/ to handle this. Ours – imaginatively again – is called /etc/logrotate.d/tiger.conf and contains in part:
/var/log/everything
{
rotate 28
daily
missingok
notifempty
delaycompress
compress
sharedscripts
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}
That says in part:
- Keep 28 old versions of the file
- Rotate it once a day
- Compress the newest-but-one file (so the most recently rotated is not compressed)
- After rotating, run
invoke-rc.d rsyslog rotateto force rsyslog to open a new log file.
The invoke-rc.d line is Debian-specific and may need to be changed for other distributions. The rest of the file can be used as-is.
Conclusion
After 15-odd years of using this technique, its value is without question. We find ourselves looking at the “everything” log more often than any other system log file.
Was This Linux Tip Useful?
Let us know in the comments below.