Process is stuck with more virtual memory than physical memory available

Question

I have a process that seems to be stuck in an interesting way. top shows this:

   PID USER       PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 38984 gergely    20   0  332m  276  184 S  0.7  0.0  18:13.16 Holmake
 79492 gergely    20   0 10.7g 6.5g  316 D  0.7 84.5 107:04.76 buildheap

buildheap uses 84.5% of physical memory and 10.7 GB of virtual memory in all. This is on a machine with 8GB physical memory assigned. But the process itself uses only 0.7% of the cpu cycles.

Following answer Linux: how to detect if a process is thrashing too much?

the 12th variable in /proc/79492/stat is 336236. Is this way too high?

$ cat /proc/79492/oom_score
333

Again this is not clear whether this is too high or not.

Does this show too much thrashing?

And, is there any other cure than asking the sysadmin to assign more phyiscal memory to the machine?

Answer 1

I wouldn't assume that memory exhaustion is the problem. If it was, I'd expect to see the process terminated by the kernel OOM (out of memory) killer, or page allocation faults in dmesg. Also bear in mind that overcommitting of memory and swapping is "normal", although there are obvious performance implications.

The easiest way to see what's really going on is to use strace to dump out the processes syscalls.

strace -p <PID> will attach strace to a running process, "-p" option is for PID of the process.

Alternatively, you can run your program directly with strace:

strace buildheap

With a bit of luck you'll be able to see what the process is waiting on, such as trying to read a file it doesn't have permission to access.