There are plenty of questions and answers about constraining the resources of a single process, e.g. RLIMIT_AS can be used to constrain the maximum memory allocated by a process that can be seen as VIRT in the likes of
top. More on the topic e.g. here Is there a way to limit the amount of memory a particular process can use in Unix?
setrlimit(2) documentation says:
A child process created via fork(2) inherits its parent's resource limits. Resource limits are preserved across execve(2).
It should be understood in the following way:
If a process has a RLIMIT_AS of e.g. 2GB, then it cannot allocate more memory than 2GB. When it spawns a child, the address space limit of 2GB will be passed on to the child, but counting starts from 0. The 2 processes together can take up to 4GB of memory.
But what would be the useful way to constrain the sum total of memory allocated by a whole tree of processes?