I'm running Gentoo Hardened AMD64 using kernel 4.3.3-hardened-r4. With my system only running some basic daemons like wpa_supplicant, cron or DHCP, and with an X session with only Windowmaker, GKrellM and xterm open, Linux starts eating more and more RAM as time goes on until after some 8-12 hours it ends up running out of RAM and throws a kernel panic. This is not a matter of Linux reporting RAM used for buffers and filesystem cache as taken, because top, htop and GKrellM both account for these cases and display how much RAM is actually taken by processes. Up until recently I thought it was linked to my Bitcoin Core client, but that was not the case (I just casually happened to run that app while my Linux system was up).
I was able to see in a few instances my RAM usage suddenly jumping back to normal while issuing a full world update (
emerge -NDu --with-bdeps=y @world), but I haven't been able to reproduce this workaround.
So far I've tried the following fixes:
- Compiling NUMA support on my kernel (by default not enabled by Gentoo's genkernel) and adding
vm.zone_reclaim_mode=1to my sysctl. Didn't work.
vm.drop_caches=1to my sysctl. Didn't work.
- Checking whether a tmpfs mount was getting full. My tmpfs mounts hardly even register over 1 MB of filesystem usage.
Evidence of this behavior can be seen in these screenshots:
Exhibit A: In which the only memory-eating processes that are running are Firefox, GKrellm and X yet Linux is eating almost 3 GB of core. Note: I didn't have my swap space enabled here (it's on an USB 3.0 external HD because my internal HD is old and slow), but even with swap enabled I still end up with an OOM kernel panic after 8+ hours of keeping Bitcoin Core running.
Exhibit B: Just in case htop and GKrellm are flawed, I double-checked with top. Same result.
Exhibit C: My tmpfs mount usage statistics, my output of
free and my content of
/proc/meminfo available here.
This post has been greatly edited to account for my most recent findings. The old post can be found on this Pastebin here.