I was greping syslog on a server for errors and noticed these lines that had nothing to do with what I was looking for but was alerting enough:
kernel: [8456851.218586] ata2.00: irq_stat 0x08000000, interface fatal error
kernel: [8456851.220076] res 50/00:00:80:45:c1/00:00:08:00:00/e0 Emask 0x10 (ATA bus error)
How can I be sure that my servers alert me with an email when similar errors occurs? I need to at least catch all drive errors (when it comes to server H/W I have to deal with power supply interruptions and HDD failures more than anything else). However anything else that has a high chance of indicating an upcoming failure is worthy of an email just as long as I don't get too much false negatives that I stop looking carefully into these emails.