I just took a look at the output of top and it showed me (amongst other processes) the following:
As once can see I have ten processes consuming approx 10GB each so 100GB in total. The computer however has only 64GB of memory as can be seen in the second line from the top. Of which currently about 22GB are used.
Now the solution to this puzzle: The test_mpi.out processes use a large amount of shared memory amongst each other. Since I have the source code I know that the actual memory consumption is about 10GB.
The computer consumes about 12GB of memory when idle, so this sums up to the reported 22GB.
What I don't understand is how top knows that only 22GB of memory are actually used. Based on all the columns displayed (VIRT,RES and SHR) top should not be able to figure this out. It would be awesome if someone could shed some light on this.
EDIT: running on redhat linux
EDIT: Thanks to Michael Homer I know now that top takes this information from /proc/meminfo. But I was rather hoping for an answer that would explain how I can determine that all the test_mpi.out processes only consume 10GB in total (instead of the 100gb suggested by naively adding the output of top). I tried looking at /proc/PID/status but I didn't find any clues on how I could determine the actual memory usage of several processes that share a large memory segment (if I wouldn't have the source code).