1

I've got a server that claims 1.7 of its 2 GB of RAM are used, but the combined memory usage of the running processes doesn't come anywhere near that figure:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1993       1898         94          0          0        199
-/+ buffers/cache:       1699        294
Swap:         2046         40       2006

$ grep VmRSS /proc/*/status | awk '{s+=$2} END {print s}'  
29080

So far, the explanations I've found blame high RAM usage on caches and buffers or the kernel's slab cache, but neither of those seems to be the reason in this case (see also /proc/meminfo below).

What else could be causing this excessive RAM usage (which often causes the system to start swapping, even with vm.swappiness=1)?
What can I do to find and perhaps mitigate the reason?


Potentially relevant information:
The system is a rented dedicated server with an Intel Atom N2800 CPU and 2 GB of RAM, as seen above. It is running Debian Wheezy (7.6) with the 3.13 kernel from wheezy-backports.
Its current /proc/meminfo:

MemTotal:        2041464 kB
MemFree:           95224 kB
Buffers:              20 kB
Cached:           204240 kB
SwapCached:        13000 kB
Active:           100280 kB
Inactive:         132036 kB
Active(anon):      15496 kB
Inactive(anon):    12580 kB
Active(file):      84784 kB
Inactive(file):   119456 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2096124 kB
SwapFree:        2054960 kB
Dirty:                64 kB
Writeback:             0 kB
AnonPages:         17208 kB
Mapped:             6888 kB
Shmem:                20 kB
Slab:              31180 kB
SReclaimable:      14628 kB
SUnreclaim:        16552 kB
KernelStack:        1848 kB
PageTables:         4556 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3116856 kB
Committed_AS:     122748 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       81792 kB
VmallocChunk:   34359611556 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      136108 kB
DirectMap2M:     1943552 kB

ps -eo pid,user,args,pmem --sort pmem, as Braiam requested:

  PID USER     COMMAND                     %MEM
    2 root     [kthreadd]                   0.0
    3 root     [ksoftirqd/0]                0.0
    5 root     [kworker/0:0H]               0.0
    7 root     [rcu_sched]                  0.0
    8 root     [rcu_bh]                     0.0
    9 root     [migration/0]                0.0
   10 root     [watchdog/0]                 0.0
   11 root     [watchdog/1]                 0.0
   12 root     [migration/1]                0.0
   13 root     [ksoftirqd/1]                0.0
   15 root     [kworker/1:0H]               0.0
   16 root     [watchdog/2]                 0.0
   17 root     [migration/2]                0.0
   18 root     [ksoftirqd/2]                0.0
   20 root     [kworker/2:0H]               0.0
   21 root     [watchdog/3]                 0.0
   22 root     [migration/3]                0.0
   23 root     [ksoftirqd/3]                0.0
   25 root     [kworker/3:0H]               0.0
   26 root     [khelper]                    0.0
   27 root     [kdevtmpfs]                  0.0
   28 root     [netns]                      0.0
   29 root     [writeback]                  0.0
   30 root     [kintegrityd]                0.0
   31 root     [bioset]                     0.0
   32 root     [kblockd]                    0.0
   35 root     [khungtaskd]                 0.0
   36 root     [kswapd0]                    0.0
   37 root     [ksmd]                       0.0
   38 root     [khugepaged]                 0.0
   39 root     [fsnotify_mark]              0.0
   40 root     [crypto]                     0.0
   45 root     [kthrotld]                   0.0
   46 root     [deferwq]                    0.0
  151 root     [khubd]                      0.0
  156 root     [ata_sff]                    0.0
  168 root     [scsi_eh_0]                  0.0
  170 root     [scsi_eh_1]                  0.0
  171 root     [scsi_eh_2]                  0.0
  172 root     [scsi_eh_3]                  0.0
  191 root     [kworker/3:1H]               0.0
  192 root     [kworker/1:1H]               0.0
  193 root     [kworker/0:1H]               0.0
  197 root     [kworker/2:1H]               0.0
  212 root     [bioset]                     0.0
  218 root     [btrfs-genwork-1]            0.0
  219 root     [btrfs-submit-1]             0.0
  220 root     [btrfs-delalloc-]            0.0
  221 root     [btrfs-fixup-1]              0.0
  224 root     [btrfs-rmw-1]                0.0
  225 root     [btrfs-endio-rai]            0.0
  226 root     [btrfs-endio-met]            0.0
  228 root     [btrfs-freespace]            0.0
  230 root     [btrfs-cache-1]              0.0
  231 root     [btrfs-readahead]            0.0
  233 root     [btrfs-qgroup-re]            0.0
  234 root     [btrfs-cleaner]              0.0
  235 root     [btrfs-transacti]            0.0
  415 root     [kworker/2:2]                0.0
  533 root     [kpsmoused]                  0.0
  619 root     [kworker/3:0]                0.0
 1383 root     [btrfs-genwork-1]            0.0
 1384 root     [btrfs-submit-1]             0.0
 1385 root     [btrfs-delalloc-]            0.0
 1386 root     [btrfs-fixup-1]              0.0
 1389 root     [btrfs-rmw-1]                0.0
 1390 root     [btrfs-endio-rai]            0.0
 1391 root     [btrfs-endio-met]            0.0
 1393 root     [btrfs-freespace]            0.0
 1395 root     [btrfs-cache-1]              0.0
 1396 root     [btrfs-readahead]            0.0
 1398 root     [btrfs-qgroup-re]            0.0
 1399 root     [btrfs-cleaner]              0.0
 1400 root     [btrfs-transacti]            0.0
 1687 root     /usr/sbin/sshd               0.0
 2087 root     [kworker/1:0]                0.0
 2218 root     [kworker/u8:0]               0.0
 2470 root     tmux                         0.0
 2473 root     -bash                        0.0
 4643 root     tmux a                       0.0
 6072 root     [kworker/u8:1]               0.0
 7126 root     [btrfs-worker-2]             0.0
14524 root     [btrfs-delayed-m]            0.0
16555 root     [kworker/0:2]                0.0
20514 root     [btrfs-flush_del]            0.0
22924 root     [btrfs-endio-2]              0.0
23031 root     [btrfs-worker-3]             0.0
26010 root     [btrfs-endio-3]              0.0
26227 root     [btrfs-endio-met]            0.0
26383 root     [btrfs-delayed-m]            0.0
26823 root     [btrfs-endio-wri]            0.0
27063 root     [btrfs-endio-met]            0.0
28637 root     [kworker/0:0]                0.0
30011 root     [kworker/2:0]                0.0
30832 root     [btrfs-flush_del]            0.0
31387 root     [kworker/1:2]                0.0
31475 root     [kworker/3:2]                0.0
31803 root     [btrfs-endio-wri]            0.0
32361 root     [kworker/1:1]                0.0
  499 root     lxc-start -d -n io           0.0
 1682 sshd     /usr/bin/dbus-daemon --syst  0.0
 2181 mail     /usr/sbin/nullmailer-send -  0.0
13533 root     -bash                        0.0
19059 root     lxc-start -n callisto -d     0.0
20446 root     /usr/sbin/sshd               0.0
20693 root     /usr/sbin/acpid              0.0
 1708 root     /sbin/getty 38400 console    0.0
 1709 root     /sbin/getty 38400 tty1 linu  0.0
 1710 root     /sbin/getty 38400 tty2 linu  0.0
 1711 root     /sbin/getty 38400 tty3 linu  0.0
 1712 root     /sbin/getty 38400 tty4 linu  0.0
 1832 root     -bash                        0.0
 2276 root     /sbin/getty 38400 tty1       0.0
 2277 root     /sbin/getty 38400 tty2       0.0
 2278 root     /sbin/getty 38400 tty3       0.0
 2279 root     /sbin/getty 38400 tty4       0.0
 2280 root     /sbin/getty 38400 tty5       0.0
 2281 root     /sbin/getty 38400 tty6       0.0
13990 root     -bash                        0.0
20478 root     /sbin/getty 38400 console    0.0
20479 root     /sbin/getty 38400 tty1 linu  0.0
20480 root     /sbin/getty 38400 tty2 linu  0.0
20481 root     /sbin/getty 38400 tty3 linu  0.0
20482 root     /sbin/getty 38400 tty4 linu  0.0
24167 root     -bash                        0.0
31031 root     -bash                        0.0
19067 root     init [3]                     0.0
 2132 root     -bash                        0.0
 4549 root     -bash                        0.0
    1 root     init [2]                     0.0
 2508 root     tmux a                       0.0
  503 root     init [3]                     0.0
 4652 root     -bash                        0.0
 7923 root     -bash                        0.0
 2130 root     sshd: root@pts/5             0.0
 1824 root     ssh io                       0.0
 2140 root     /usr/sbin/cron               0.0
  504 root     udevd --daemon               0.0
  505 root     udevd --daemon               0.0
 1826 root     sshd: root@pts/0             0.0
 4544 root     sshd: root@pts/0             0.0
  381 root     udevd --daemon               0.0
16820 root     /usr/sbin/vnstatd -d         0.0
14381 root     /usr/sbin/sshd               0.0
 2224 101      /usr/bin/dbus-daemon --syst  0.0
 8182 ntp      /usr/sbin/ntpd -p /var/run/  0.0
 2497 nst      bash                         0.0
 6459 root     -bash                        0.0
 2382 root     /usr/sbin/console-kit-daemo  0.0
 2472 root     tmux                         0.0
 3074 root     ps -eo pid,user,args,pmem -  0.0
17115 root     /usr/sbin/rsyslogd -c5       0.0
 6149 root     htop                         0.0
 3075 root     view -                       0.1
  911 root     munin-node                   0.1
 2231 snmp     /usr/sbin/snmpd -Lsd -Lf /d  0.1
13988 root     tmux                         0.1
26168 root     -bash                        0.1
27332 root     -bash                        0.1
  628 root     -bash                        0.1
  • Add to the question the output of ps -eo pid,user,args,pmem --sort pmem. – Braiam Aug 6 '14 at 18:32
  • @Braiam: Done.⁠ – n.st Aug 6 '14 at 18:41
  • Weird. Is this a persistent problem? What does this server do? Could you try switching to a different kernel? – Faheem Mitha Aug 6 '14 at 18:56
  • @FaheemMitha The server is running two LXC containers, both of which are idle at the moment. I ran tightvncserver and Chromium in one of them earlier, but with most of the host's RAM in use, that's currently too slow to be useful. I'd rather not switch to a different kernel since I don't have KVM access to the server, so it's always a pain in the backside if a kernel update breaks it (I had that pleasure with 3.14)… – n.st Aug 6 '14 at 19:02
  • Ok. Part of the motivation for this was that (if the box had been hacked) the hacker could be hiding running processes, but I suppose that is unlikely. – Faheem Mitha Aug 6 '14 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.