Best practices for maintaining error logs (system, device driver & application)

Question

I have many Pandaboards with Ubuntu armhf and one PC server networked with Wi-Fi. On each embedded board I have a real-time application running constantly.

I don't know how to reasonably maintain the logs, because I need to monitor so many things:

1. System faults (filesystem & overheating & misc)
2. Device driver faults (wi-fi & usb)
3. Application shutdowns (crashes & intended ones)

Ideas I have so far:

1. Find relevant system faults by filtering syslog, e.g. severity levels 0-3. How to do this?
2. Device driver problems should be found from syslog as well.
3. Currently I'm logging cout and cerr to separate files. This does not help if the app crashes. I would guess running in debug mode takes too much processing power and the system won't be real-time any more.
4. Instead of having daily log rotation, I would prefer having one syslog for each time computer is started up OR until the log is too big. Also, I would prefer to rotate the application logs each time the app starts.
5. All severe problems could be piped into one file that is easily readable from the server.

Any ideas to efficiently maintain the error logging are welcome. Currently, I'm using Fabric python library for transferring diagnostics info and updates between the server PC and the Pandaboards.

Answer 1

The system logger is called syslog, although your Ubuntu may or may not be using a more complex variant called rsyslog. The easiest way to tell is via ls /etc | grep syslog. They both have their main configuration file there and probably also a .d directory from which further configuration is sourced. You can feed the system logger from an application (see, e.g., man logger and man 3 syslog), although unless you want a lot of awkward clutter, this is best reserved for significant errors.

There are many introductions to and tutorials for syslog online. Rsyslog is intended to be syslog compatible, and additional documentation is available through their site. Default configuration usually filters messages into various files in /var/log based on facility and severity. There is likely also one file which collects a copy all all messages (e.g. /var/log/syslog) and there may be other forms of overlap as well. Rsyslog offers the further ability to filter messages based on content (so, e.g., you can match against tags usually appended for specific applications).

WRT to log rotation, the relevant application is logrotate (see man logrotate) which is usually run by cron -- it is not a daemon. Controlling the frequency, etc., should be done via its configuration (usually /etc/logrotate.conf which sources other files in /etc/logrotate.d) and cron's. Note that logrotate does not use a default configuration file, so you may want to look at how it is invoked in the relevant crontab.