1

Text analytics programming today. My program is going to write about 45 million small temporary files to /dev/shm and their size estimate is 177.0 GB. I'm going to delete them soon after the next step in my processing. So the problem is that at 128GB physical RAM on my host, I won’t have enough physical RAM.

OK, question: Will my swap kick in automatically when I write to /dev/shm, because I have 239GB of total swap plus 128GB RAM so that’s plenty, if so.

Second question: I currently have 100G allocated to /dev/shm. Will I need in advance, to over-allocate (beyond physical RAM) my /dev/shm up to 177G (or rather 190G to be safe)? Or do I not need to set my /dev/shm size in advance to 190G because the swap will automatically take care of the usage beyond 100G I currently have allocated?

By the way my inodes is already 60 million which is plenty for my 45 million estimated requirement. THis is good. Also BTW I found that linux ext4 malfunctions at around 10.5 million files on disk, repeatably, but not deterministically. It seems to be a timeout problem due to design of something going very slowly as far as python file writing is expecting, probably a linear-time-complexity algo for table insertion of inodes of the filesystem which just takes too long for python file writing infrastructure to create any more files, despite the supposed nominal 2 billion file count limit. SSD and spinning disk behave the same here, no difference. inodes were fine and free space was fine, just always got OSErrors on true filesystems (which is why I'm going to /dev/shm) due to timeouts breaking the system.

This host:

$ uname -a
Linux ga-HP-Z820 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64
ga@ga-HP-Z820:/mnt/fastssd/bot_subreddit_recom/tmp$ df -i /dev/shm
Filesystem       Inodes IUsed    IFree IUse% Mounted on
tmpfs          60000000     1 59999999    1% /dev/shm
ga@ga-HP-Z820:/mnt/fastssd/bot_subreddit_recom/tmp$ df -BG /dev/shm
Filesystem     1G-blocks  Used Available Use% Mounted on
tmpfs               100G    0G      100G   0% /dev/shm
ga@ga-HP-Z820:/mnt/fastssd/bot_subreddit_recom/tmp$ 

Additional info:

Here IT says

If you oversize your tmpfs instances the machine will deadlock since the OOM handler will not be able to free that memory

. So I guess I won’t do that!!! https://www.cyberciti.biz/files/linux-kernel/Documentation/filesystems/tmpfs.txt

Therefore my decision is to run an experiment to find out: Let swap take care of things. Just run it as is right now. Let my files exceed the allocated size of the /devshm, and let swap do the rest.

1

I just went ahead and see what happens, when you write too many files to /dev/shm.

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/fastssd/bot_subreddit_recom/write_user_docs.py", line 85, in <module>
    f.write(subreddits)
OSError: [Errno 28] No space left on device
inFile: RC_2018-01-02
outDir: tmp
outputToScreenOnly: 0
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/fastssd/bot_subreddit_recom/write_user_docs.py", line 85, in <module>
    f.write(subreddits)
OSError: [Errno 28] No space left on device

It “ran out of space” after 33 days of input files, as shown above.

>>> 36.*33/12
99.0

So an estimated 99GB -- which is very close to 100GB -- of files, is where the “no space left” error happened. This is exactly the size of the /dev/shm device which I had set to be 100G. This means swap was not really able to be used when it was needed. This has demonstrated that unfortunately, /dev/shm simply runs out, and does not let you start to use swap as virtual memory to keep going, when there’s a shortage. So now we know thanks to this experiment. You just need real physical RAM for /dev/shm to work, and virtual RAM aka swap is not helpful for /dev/shm at all.

I have posted this answer to share with others who may benefit from the information gained.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.