1

I'm running Gentoo Hardened AMD64 using kernel 4.3.3-hardened-r4. With my system only running some basic daemons like wpa_supplicant, cron or DHCP, and with an X session with only Windowmaker, GKrellM and xterm open, Linux starts eating more and more RAM as time goes on until after some 8-12 hours it ends up running out of RAM and throws a kernel panic. This is not a matter of Linux reporting RAM used for buffers and filesystem cache as taken, because top, htop and GKrellM both account for these cases and display how much RAM is actually taken by processes. Up until recently I thought it was linked to my Bitcoin Core client, but that was not the case (I just casually happened to run that app while my Linux system was up).

I was able to see in a few instances my RAM usage suddenly jumping back to normal while issuing a full world update (emerge -NDu --with-bdeps=y @world), but I haven't been able to reproduce this workaround.

So far I've tried the following fixes:

  1. Compiling NUMA support on my kernel (by default not enabled by Gentoo's genkernel) and adding vm.zone_reclaim_mode=1 to my sysctl. Didn't work.
  2. Adding vm.drop_caches=1 to my sysctl. Didn't work.
  3. Checking whether a tmpfs mount was getting full. My tmpfs mounts hardly even register over 1 MB of filesystem usage.

Evidence of this behavior can be seen in these screenshots:

Exhibit A: In which the only memory-eating processes that are running are Firefox, GKrellm and X yet Linux is eating almost 3 GB of core. Note: I didn't have my swap space enabled here (it's on an USB 3.0 external HD because my internal HD is old and slow), but even with swap enabled I still end up with an OOM kernel panic after 8+ hours of keeping Bitcoin Core running.

enter image description here

Exhibit B: Just in case htop and GKrellm are flawed, I double-checked with top. Same result.

enter image description here

Exhibit C: My tmpfs mount usage statistics, my output of free and my content of /proc/meminfo available here.

This post has been greatly edited to account for my most recent findings. The old post can be found on this Pastebin here.

|improve this question
  • 1
    This, maybe? linuxatemyram.com – MatthewRock Mar 17 '16 at 10:53
  • Not my case. htop and GKrellm both ignore caches and buffers when calculating claimed RAM. – RAKK Mar 18 '16 at 10:55
  • 1
    Try adding vm.zone_reclaim_mode=1 to your sysctl.conf – ugjka Mar 21 '16 at 15:49
  • I get sysctl: cannot stat /proc/sys/vm/zone_reclaim_mode: No such file or directory. Given that I use Gentoo, this usually means I haven't compiled this feature. However, I couldn't find anything neither on make menuconfig nor on the internet. – RAKK Mar 21 '16 at 17:14
  • Would also like to clarify that I did this manually instead of adding this parameter to sysctl.conf, I used sysctl vm.zome_reclaim_mode=1 on a ROOT shell. – RAKK Mar 22 '16 at 1:11
1

Do you have an SHM based mount, such as memory-backed /tmp or /var/tmp? It's possible that temporary files are being generated which consume ram even after the process exits. These files will remain in ram until they are deleted or the system is restarted. Check your mounts in /etc/fstab and also with mount for tmpfs entries.

Also check your log rotation, as it might be creating large files in a temp directory. It might be worth clearing the journal if you're using systemd. e.g:

journalctl --vacuum-size=500M
|improve this answer
  • As far as I remember, my tmpfs mounts were pretty much empty. Will confirm later today once I return home. – RAKK Mar 18 '16 at 19:04
  • As for log rotation, I use OpenRC (default init in Gentoo). – RAKK Mar 18 '16 at 19:04
  • I switched from OpenRC to systemd 2 years ago for gnome3. I miss it sometimes. Are you using syslog-ng? – Thomas Zwaagstra Mar 18 '16 at 19:19
  • Please post the output from free as well when you get a chance – Thomas Zwaagstra Mar 18 '16 at 19:20
  • Oh and the output of /proc/meminfo would be helpful – Thomas Zwaagstra Mar 18 '16 at 19:23
1

To sum it up:

  • After using BitCoin client, it starts eating up you ram to the point where it crashes
  • It doesn't return memory(until you do something strange)

The first one looks like a classic memory leak. You can check the performance and memory management of the program by using valgrind, but it will slow it down considerably.

The second one could be the offspring of the first problem. I do not know why is this happening, but I can only guess that because of the problems with the memory(or large memory consumption, or maybe some other bug - e.g. process stuck on D state?). Since other applications don't show the same behavior, I'd guess that the problem is the bitcoin software, and not your system.

Therefore, everything we do to fix it would be a hack. There might be a successful hack, but it's still not the best way. If you have an access to the source code, and know a bit of programming, you can try running some static code analyzer to see if there are any 'simple' bugs to fix. You can try debugging its memory management with valgrind. If you don't have any of these(code/skills), last thing you can do is give feedback to the devs - probably some bugtracker, forum or mailing list. This way someone will look into it, and confirm (and hopefully fix) the problem.

|improve this answer
  • Time to consider an actual bug report? I never thought I'd ever get to that point... – RAKK Mar 23 '16 at 16:28
  • 1
    @RAKK Bugs happen. I've filed many bugs report throughout my life, it only sounds scary. Often it's also a good idea to get in touch with someone who might know the situation better - so even if it's not a bug, you are likely to get an answer. You don't need to spend that much time too - a win-win situation. – MatthewRock Mar 23 '16 at 18:20
0

So after almost two months of puzzling over this question, I decided to ask around how to enable that vm.zone_reclaim_mode sysctl option and play a little with different values, and lo and behold -- problem solved.

Solution:

  1. Enable CONFIG_NUMA on my kernel's configuration and rebuild it.
  2. Put vm.zone_reclaim_mode = 7 on sysctl.conf

Now my system can finally stay in one piece for over 24 hours.

I'm a bit afraid of reduced performance since the kernel documentation says both NUMA and such an aggressive zone reclaim setting could probably slow things down, but for now my system finally works.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.