There is almost certainly a better way of doing this, probably through
limits.conf but I don't know it so here's a dirty hack.
This command will kill all processes owned by the user
terdon that are using more than 3G of RAM:
ps -u terdon -o vsize= -o pid= |
while read mem pid; do [ $mem -gt 3145728 ] && kill $pid;
ps -u terdon -o vsize= -o pid= command will print the PID and memory used (1024-byte units) by each of the processes owned by
terdon. Obviously, you should adapt this to use your friend's user name. The output of the
ps command is then passed through a while loop that reads the memory usage and pid and then kills the PID if it is using more than 3145728K of memory.
If you now add this as a crontab run by root, it will be run every minute and kill those processes that are using more RAM than you want to allow. So, add this line to
* * * * * root ps -u terdon -o vsize= -o pid= | while read mem pid; do [ $mem -gt 3145728 ] && kill $pid; done
As I said, this is inelegant and just an ugly hack but it might be enough.