ClamAV holds the search strings using the classic string (Boyer Moore) and regular expression (Aho Corasick) algorithms. Being algorithms from the 1970s they are extemely memory efficient.
The problem is the huge number of virus signatures. This leads to the algorithms' datastructures growing quite large.
You can't send those datastructures to swap, as there are no parts of the algorithms' datastructures accessed less often than other parts. If you do force pages of them to swap disk, then they'll be referenced moments later and just swap straight back in. (Technically we say "the random access of the datastructure forces the entire datastructure to be in the process's working set of memory".)
The datastructures are needed if you are scanning from the command line or scanning from a daemon.
You can't use just a portion of the virus signatures, as you don't get to choose which viruses you will be sent, and thus can't tell which signatures you will need.
Here's the memory used on a 32-bit machine running Debian Wheezy and it's clamd.
Private + Shared = RAM used Program
281.7 MiB + 422.5 KiB = 282.1 MiB clamd
Edit: I see someone suggests setting the resident set size. If this succeeds then having a resident set size less than the working set size will lead to the process thrashing to and from swap. This will lower the entire system performance substantially. In any case the Linux manual page for setrlimit(RLIMIT_RSS, ...) says that setting the resident set size is no longer supported and never had any effect on processes which chose not to call madvise(MADV_WILLNEED, ...).