1

Just within the past day, my Arch Linux system has begun eating memory. This seems to begin when the system is powered on, and I cannot find a way to stop it. The symptom is that the reported memory usage begins slowly rising, without any concomitant rise in the memory attributed to any one process. This finishes, within eight hours or so of boot, with the system freezing up due to swap-thrashing, OOM-killing every process, and finally hitting kernel panic with "out of memory and no killable processes".

An odd aspect is that memory usage reports are not uniform across tools; top and free report less memory used than htop at any given time, though all still report far more than is used by processes. The only obvious connection I can make to the issue's beginning is that yesterday is the first time I ran VMware Player on this OS install; however, the system has been rebooted multiple times since without ever running it, and none of its kernel modules are loaded after reboot, so I'm very unsure how it could be causing this problem.

Technical data:

# free
              total        used        free      shared  buff/cache   available
Mem:        4051344     2966128      335004        3472      750212      817888
Swap:       5464060           0     5464060

# cat /proc/meminfo 
MemTotal:        4051344 kB
MemFree:          379164 kB
MemAvailable:     861696 kB
Buffers:           51988 kB
Cached:           428512 kB
SwapCached:            0 kB
Active:           828600 kB
Inactive:         341056 kB
Active(anon):     689964 kB
Inactive(anon):     2952 kB
Active(file):     138636 kB
Inactive(file):   338104 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       5464060 kB
SwapFree:        5464060 kB
Dirty:               184 kB
Writeback:             0 kB
AnonPages:        689444 kB
Mapped:           215500 kB
Shmem:              3472 kB
Slab:             265636 kB
SReclaimable:      35540 kB
SUnreclaim:       230096 kB
KernelStack:        4304 kB
PageTables:        10368 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     7489732 kB
Committed_AS:    1746596 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      321932 kB
VmallocChunk:   34358947836 kB
HardwareCorrupted:     0 kB
AnonHugePages:    192512 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      136576 kB
DirectMap2M:     4057088 kB
DirectMap1G:           0 kB

The only numbers from /proc/meminfo that look suspicious are the VmallocTotal and VmallocChunk numbers, but those are far bigger than my physical memory and swap combined, so I don't know the significance.

  • This is a generalization and a big prejudice on my part but when things go haywire with memory, first thing I look into, are the apps written in Java. I am not sure what your situation is, but if this refers to your workplace and if you are running java based application servers, likes of weblogic, tomcat, etc or some other esoteric application, some outsourcing outfit whipped up in your off-shore development house, I would look into those. I'm sorry in advance if this is an absolute BS, but this is my personal experience. – MelBurslan Feb 2 '16 at 4:03
  • It's my home machine, and I'm reasonably sure Java is not even installed. – Tom Hunt Feb 2 '16 at 4:09
  • java may not be installed but an application written in Java, then compiled are still susceptible to the same behavior, as most of them do not do a good housekeeping as far as cleaning up shared memory goes. What is the top memory user application in this case ? – MelBurslan Feb 2 '16 at 4:12
  • The top memory user is firefox; it's taking up about 9%. Memory used, meanwhile, is around 97%. Nothing else is above 2%. – Tom Hunt Feb 2 '16 at 5:03
  • 2
    Look at the /tmp dir vor similar mounted tmpfs. There might be a process filling one of them, and as they are located in ram, filling it too. – bvolkmer Feb 2 '16 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.