email alerts when serious system errors occur

Question

I was greping syslog on a server for errors and noticed these lines that had nothing to do with what I was looking for but was alerting enough:

kernel: [8456851.218586] ata2.00: irq_stat 0x08000000, interface fatal error
kernel: [8456851.220076]          res 50/00:00:80:45:c1/00:00:08:00:00/e0 Emask 0x10 (ATA bus error)

How can I be sure that my servers alert me with an email when similar errors occurs? I need to at least catch all drive errors (when it comes to server H/W I have to deal with power supply interruptions and HDD failures more than anything else). However anything else that has a high chance of indicating an upcoming failure is worthy of an email just as long as I don't get too much false negatives that I stop looking carefully into these emails.

Answer 1

I use logcheck to scan my log files. It scans new entries once an hour and emails me any unknown lines as well as known lines which are classified inclusion in the report.

There are also tools which will watch your log files and email you when they certain content.

Answer 2

I use the ommail module in rsyslog. Configuration is straightforward, you can use multiple smtp servers for sending your alert emails (as a failover):

$template mailSubject,"EMERGENCY:"
$template mailBody," %$NOW% %timereported:8:15%  %programname%: %msg%\n"

$ActionMailSMTPServer                   mail1.example.com
$ActionMailFrom                         rsyslog@your-server.com
$ActionMailTo                           martin@yahoo.com
$ActionMailSubject                      mailSubject
$ActionExecOnlyOnceEveryInterval        60
*.emerg                                 :ommail:;mailBody

# begin backup action, carried out if primary fails
$ActionExecOnlyWhenPreviousIsSuspended  on
$ActionMailSMTPServer                   mail2.example.com
$ActionMailFrom                         rsyslog@your-server.com
$ActionMailTo                           martin@yahoo.com
$ActionExecOnlyOnceEveryInterval        60
&