2

I get the feeling I'm going insane, or tmpfs is very broken for long term use.

I have a workload that very rapidly creates and unlinks files in /dev/shm/[some directory tree]. Linux Slab usage (in size-64 and size-128) increases linearly with inodes allocated/unlinked and never drops (the memory is listed an unreclaimable via meminfo, and slabinfo shows many millions of active objects).

This memory is never reclaimed, and if allowed to continue, OOM. The only fix is unmounting and remounting /dev/shm.

Another user asked this question a few years ago, but the answer did not actually cover the problem in question (operation in /dev/shm causes overflow).

Is this simply a design decision for tmpfs or is there something else going on here? It feels terribly broken that inodes would never be freed once allocated.

Timeline: Process creates 5 million files, one at a time, and unlinks each immediately after creation. All user processes killed at this point. Memory usage is as if 5 million inodes are still in /dev/shm, although df -i and df -h report that /dev/shm is essentially empty. Further iterations of the process loop linearly increase memory usage until the system is totally out of memory and OOMs.

EDIT: For anyone stumbling on this later, this seems to be an artifact of the older kernel I was running (SLES 11, 2.6.32-something). Newer kernels cannot reproduce the problem.

  • That other answer is not correct, as it ignores the fact that his system did run out of memory over time. Perhaps his workload was the culprit, but in my case, I can state it absolutely is not. I'm performing a mknod immediately followed by an unlink. The file is never open, so it's not being held. Further, the process that created/deleted the files is no longer running when I'm poking at the memory usage (there are zero user processes). Memory usage grows until the system is out of memory and OOMs, even with reasonable limits on /dev/shm (in this case, 16 GB and 5 million inodes). – Waco Sep 14 '16 at 20:06
  • Timeline: Process creates 5 million files, one at a time, and unlinks each immediately after creation. All user processes killed at this point. Memory usage is as if 5 million inodes are still in /dev/shm, although df -i and df -h report that /dev/shm is essentially empty. Further iterations of the process loop linearly increase memory usage until the system is totally out of memory and OOMs. – Waco Sep 14 '16 at 20:11
  • Give us the kernel version (I assume 4.7.2 but I wanna make sure) and if it has any non-common options. Also, remember that tmpfs will simply respect its size= parameter, if the machine has no swap, is using 51% of its memory, and then you fill a tmpfs that uses default size= you will end out of memory. – grochmal Sep 14 '16 at 20:19
  • 2.6.32, SLES 11. No swap, it's a Cray supercomputer. :) Please note that when I say I run out of memory, I'm talking about the entire OS. I am not filling the filesystem in question, it is never more than 1% used in terms of space or inodes. No uncommon options for /dev/shm, just a 16 GB limit + 5 million inodes. Both comfortably fit in the 32 GB of memory on the node, and this is the only process running. You can reproduce it very quickly with a simple script that loops with a touch [unique filename] and unlink [unique filename]. – Waco Sep 14 '16 at 20:29
  • Wait, I'm confused now. 5 million inodes is some 0.6GB, therefore a default (default mount options) tmpfs on a machine with 32GBs shall never free that memory it shall just keep that in cache. I'd try echo 3 > /proc/sys/vm/drop_caches and see of the memory is freed by that. Also, it is good practice here to edit the question to add the extra info (i.e. instead of comments as we are doing). – grochmal Sep 14 '16 at 20:43
2

For the sake of clarity I'm adding a more-or-less scripted test of what we talked about in the comments. This is on kernel 4.7.2 where the issue does not happen either:

$ cd /dev/shm
$ free
              total        used        free      shared  buff/cache   available
Mem:        1794788      673948      873668       19300      247172      963316
Swap:       2097148           0     2097148
$ for i in `seq 100000`; do touch node$i; done
$ ls -1|wc -l  # oops, there are extra three pulseaudio files here
 100003
$ free
              total        used        free      shared  buff/cache   available
Mem:        1794788      738240      811944       19300      244604      890184
Swap:       2097148           0     2097148

OK, we get the memory footprint. But rm clears it

$ rm node*
$ free
              total        used        free      shared  buff/cache   available
Mem:        1794788      671484      896524       19300      226780      965884
Swap:       2097148           0     2097148

The match is not perfect 'cause I cleaned some caches in the meantime. But the amount of free memory and memory in cache is the same at the beginning and end of this little experiment.

Therefore yes, the issue happen only in an old kernel version. Which would indicate that there was a bug but it has been fixed already.

2

Looks like this is just a bug in the old kernel that runs on this particular machine. I cannot reproduce it on a newer RHEL 6 machine with the most recent kernel patches.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.