2

I have a VM that's having memory issues. One of the tasks I'd like to run in it is crashing with out-of-memory errors.

However, when it crashes, the system still stays memory starved. I'm not sure if this is just a process I'm missing, or an actual bug (this is in hyper-v, with the new kernel extensions that allow memory-ballooning for linux hosts, so it could very well be a genuine kernel bug).

durr@sqlbox:~$ free -h
             total       used       free     shared    buffers     cached
Mem:          3.1G       2.6G       541M        88K       7.4M        39M
-/+ buffers/cache:       2.5G       588M
Swap:         1.0G       6.2M       1.0G

Free tells me that it's not just cached, there is actually something that appears to be live taking up 2.6G of memory.

However, looking through the output of PS sorted by virtual size is not enlightening:

durr@sqlbox:~$ ps -e ax -o pid,vsz,comm | sort --numeric-sort --key=2
[ ... snip ... ]
  PID    VSZ COMMAND
   96      0 rcuob/23
   97      0 rcuob/24
   98      0 rcuob/25
   99      0 rcuob/26
 1124   4368 acpid
59863  10016 ps
 1031  15668 upstart-file-br
 1047  15820 getty
 1050  15820 getty
 1055  15820 getty
 1056  15820 getty
 1058  15820 getty
 1167  15820 getty
 1023  15920 upstart-socket-
 1076  19140 atd
 1099  19188 irqbalance
  428  19476 upstart-udev-br
59864  21860 sort
59267  22644 bash
59234  22664 bash
59280  22808 bash
 1075  23656 cron
59261  26928 screen
59279  27380 htop
59262  28472 screen
    1  33776 init
  749  39240 dbus-daemon
  816  43452 systemd-logind
  432  51348 systemd-udevd
 1090  61364 sshd
  871 255844 rsyslogd
59184 269028 sshd
59233 269028 sshd

So the biggest memory consumer is sshd, and it's using..... 269K? Where is all my memory going?

Looking into /proc/meminfo shows:

durr@sqlbox:~$ cat /proc/meminfo
MemTotal:        3266904 kB
MemFree:          554228 kB
Buffers:            7596 kB
Cached:            41104 kB
SwapCached:         3032 kB
Active:            32552 kB
Inactive:          34292 kB
Active(anon):      13224 kB
Inactive(anon):     5008 kB
Active(file):      19328 kB
Inactive(file):    29284 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1044476 kB
SwapFree:        1038084 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         15656 kB
Mapped:             9196 kB
Shmem:                88 kB
Slab:              33488 kB
SReclaimable:      14532 kB
SUnreclaim:        18956 kB
KernelStack:        2352 kB
PageTables:         2748 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2677928 kB
Committed_AS:      56148 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       34360 kB
VmallocChunk:   34359695660 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       44456 kB
DirectMap2M:     3491840 kB

Obviously, it appears that something has Vmalloced an enormous amount of memory, but I'm not sure if that's relevant.

5

It may be possible that your program used shared memory and didn't clean that up.

There are three variants of shared memory on linux:

1.) POSIX shared memory (the one implemented by glibc) is accessible via files on the tmpfs pseudo-filesystem and are usually mounted by the system on places like /dev/shm, /run, /run/shm or /run/lock. The best way to find out for sure is to enter mount | grep -E '^tmpfs' (more portable) or grep -E '^tmpfs' /proc/mounts (best for linux) in a shell.

Note that it is possible for a process to call unlink() on a mmap()'ed file in shared-memory, rendering the file inaccessible via filename (e.g. one needs a previously allocated file handle to continue accessing it). However, unlink()ed files are usually deleted upon exiting of all processes that had it open()'ed - maybe if your program finished, there still is another process still holding a handle to it.

2.) SysV IPC shared memory, which is not visible via the tmpfs pseudo-filesystem, but rather via /proc/sysvipc/shm, linux syscalls (only if you are a hacker), its libc wrapper or, most recently, via ipcs -m -p -[tclu]. You need to find a matching process ID in any of these lists, and then inspect that process further.

3.) Anonymous mapped shared memory, in which case the memory is not backed by any file, but rather the memory is initially shared between a process and all of its children. AFAIK such anonymous mapped shared memory is freed when the process that mmap()'ed it exit()s. So, in case your program terminated and all of its child processes did as well, they shouldn't AFAIK further occupy any memory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.