If you are concerned about system integrity, then selinux or grsecurity (or the various similar security packages) are very powerful. Unfortunately, mastering their policies is far from trivial. (Any decent distro that includes SELinux will have predefined policies for all kinds of things, though.) Grsecurity policies are easier to create but still require some effort. Grsecurity has the big advantage over SELinux that it comes with several system hardening measures, like pax, which provides quite rigorous memory protection. On the downside, Grsecurity is not officially part of Linux and never will be (for, um, political reasons) and thus only few distros provide integration of Grsecurity.
My personal view: The whole concept of AV is entirely rotten because they are - in essence - nothing more than giant black lists that need to be updated frequently. Because of this they grow ever larger and don't protect you from 0-day-exploits. Personally I believe in encapsulation and containment, which is what SELinux, Grsecurity, etc. achieve.
IDS/IPS is useful to some degree, as long as you can keep it simple (like using iptables, fail2ban, or aide). "High-end" IDS/IPS work like AV and thus my view applies for them as well.