I have many Pandaboards with Ubuntu armhf and one PC server networked with Wi-Fi. On each embedded board I have a real-time application running constantly.
I don't know how to reasonably maintain the logs, because I need to monitor so many things:
- System faults (filesystem & overheating & misc)
- Device driver faults (wi-fi & usb)
- Application shutdowns (crashes & intended ones)
Ideas I have so far:
- Find relevant system faults by filtering syslog, e.g. severity levels 0-3. How to do this?
- Device driver problems should be found from syslog as well.
- Currently I'm logging cout and cerr to separate files. This does not help if the app crashes. I would guess running in debug mode takes too much processing power and the system won't be real-time any more.
- Instead of having daily log rotation, I would prefer having one syslog for each time computer is started up OR until the log is too big. Also, I would prefer to rotate the application logs each time the app starts.
- All severe problems could be piped into one file that is easily readable from the server.
Any ideas to efficiently maintain the error logging are welcome. Currently, I'm using Fabric python library for transferring diagnostics info and updates between the server PC and the Pandaboards.