I'm currently running a rebuild on a btrfs NAS (~20TB) that had a drive failure. It's going to take a few days. My problem is about once every 24 hours the RAM filled up and the kernel starts killing off processes until the whole system dies.

htop shows this as user process memory usage but no user processes show any significant memory usage:

enter image description here

free shows this as buffered/cached ram:

# free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        923M        1.0G         16M         13G        5.0G
Swap:          7.9G          0B        7.9G

I finally tracked it down with slabtop

As you can see bio has an insane amount of slab usage. Running sync && echo 3 > /proc/sys/vm/drop_caches clears up the dentry usage but does not affect bio at all.

This NAS is running Rockstor which is based on CentOS. The machine in question is on kernel 4.4.5-1.el7.elrepo.x86_64. There is nothing of note in kernel messages until the RAM usage gets to the point where processes start getting killed off.

Is this a memory leak bug in BTRFS or the kernel? If this is a bug that needs to be reported, how to I determined which package to report it to?

Is there a way to clear the cache usage by bio so I can complete this RAID rebuild without rebooting the system once a day when it crashes from running out of RAM?

And just out of morbid curiosity, if this is kernel caching why does htop report it as user process memory usage (green bar)?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.