This question already has an answer here:
Occasionally some processes on my GNU/Linux desktop (such as gv and gnash) use up the physical memory and cause thrashing. Since these processes aren't important, I want them to be automatically killed if they use too much memory.
I think the /etc/security/limits.conf file and the -v option could be used for this. The question is whether it limits the amount of available memory per process of a particular user, or the sum for all the processes of a user. Also I would like to ask how to make change to that file in effect without rebooting.