you could limt memory usege with /etc/security/limits.conf
in this file you put :
domain type item value
domain is the
hard is limt that cannot be exceeded under any circumstances.
item is the item field specifies what type of item is being limited. Examples
include core (the size of core files), data (the size of a program’s data area), fsize (the size
of files created by the user), nofile (the number of open data files), rss (the resident set size),
stack (the stack size), cpu (the CPU time of a single process in minutes), nproc (the number
of concurrent processes), maxlogins (the number of simultaneous logins), and priority (the
process priority). The data, rss, and stack items all relate to memory consumed by a pro-
gram. These and other measures of data capacity are measured in kilobytes.
value is relate to the
item field that you have chose, for example if you have chose
cpu in the
item field and put
2 in the
value then in case the
domain (you group of VirtualBox) is take more than
2 of the cpu time it will be terminated.
You can use
ulimit too, but it restricted to the bash shell only.
Hope this help you in you goal, although it isn't done with the method you asked.