Here is a simple experiment I tried:
Given a task called "sim.exe" which doing a model simulation, I then use MPI to launch x "sim.exe" simultaneously on one node (shared memory system). I have tried four different runs with x to be different value (e.g., 1, 4, 8, 16). Then I check the memory usage through the PBS report "memory used" and "Vmem used". I observed that for these different runs the "memory used" and "Vmem used" keep the same not changed with "mem" = 8,432 KB and "vmem" = 489,716 KB.
My understanding of the "mem" and "vmem" is the memory resources used by the job, according to the post About mem and vmem and Actual memory usage of a process. Then why the "mem" and "vmem" keep the same even through the tasks of the jobs increased with x-fold?
All these Jobs are submit through PBS job scheduler. For each jobs all the cores and RAM are requested when submitting the job with
#PBS -l select=1:ncpus=24:mem=96GB
Update for this question:
I have tested
threading in python as a replacement of
mpi to launch x "sim.exe" simultaneously. So I started x thread and in each thread using subprocess to call the "sim.exe" model simulation. I again tried four experiments with x=1, 4, 8, 16. I observed that the "mem" and "vmem" used by the job is increasing linearly with the increase of x, which is close to what I would expect.
So, is it possible that PBS did not count the "mem" and "vmen" correctly? it seems that PBS only count the memory usage of only one instance.