There is something kind of what you describe: there is a feature to limit the amount of RAM used by a process (RAM, as opposed to virtual memory). The
RLIMIT_RSS limit sets an upper bound a program's resident set size, i.e. the part of the memory of that process which is resident in memory (as opposed to swapped out). However, it is not implemented on Linux.
RLIMIT_RSS existed in some old Unix systems (BSD only?) but has been dropped from many modern systems. On Linux, it existed only on the 2.4 series (and even there didn't fully work). On Solaris it was dropped at some point in the distant past (when
Solaris SunOS switched from BSD to System V, maybe?). FreeBSD seems to still have it.
I don't know why
RLIMIT_RSS was removed. Alan Cox had this to say in 2006:
The original mm for Linux didn't enforce it as it did not
have any way to track RSS that was not computationally expensive. The
current mm is probably capable of implementing RSS limits although
questions then still remain about what effect this has if a user sets a
low RSS limit and then causes a lot of swap thrashing.
If you can see a way to implement it efficiently then go for it.
Since then the topic has come up several times but as far as I know no patch was accepted to the Linux kernel.
One reason that enforcing a limit on physical memory usage for a process is that it's hard to define how much physical memory a process is using. Ok, count its stack and heap (the part that's not swapped out). What about memory-mapped files? To measure the physical memory usage of a process, you have to count the cache used by the files that it maps. What about shared libraries and other shared mappings? If they're used by a single process then obviously they should be counted against it — but when they're used by multiple processes, it's hard to know which process is using which part.
It doesn't make all that much sense to limit physical memory usage of a single process. Given that resource limits are inherited, each child would be allowed to use as much physical memory. It makes a little more sense to allow a limit on physical memory for a set of processes. Linux has this feature built into cgroups, a partial virtualization system where a process and its descendants run in a container which can have its own filesystem root, its own network controllers, its own resource limits, etc. A cgroup's memory usage can be limited with the
memory.limit_in_bytes parameter (
memory.memsw.limit_in_bytes controls the use of RAM plus swap). The documentation warns that pages shared between groups are assigned in a somewhat arbitrary way (“Shared pages are accounted on the basis of the first touch approach. The
cgroup that first touches a page is accounted for the page.”). There may be other ways in which the limit isn't strictly enforced.
The other part of what you're asking for is to have a dedicated swap file for a process. This would require considerable complexity in the kernel. Remember that pages can be shared; which swap file do you use if the two processes have different swap space assigned to them? On the other hand, the feature can be implemented fairly easily from the process side: map a file to memory. For shared pages, there's still a single file. As a side benefit, different areas of a process can use different “swap space”.