Text analytics programming today. My program is going to write about 45 million small temporary files to /dev/shm and their size estimate is 177.0 GB. I'm going to delete them soon after the next step in my processing. So the problem is that at 128GB physical RAM on my host, I won’t have enough physical RAM.
OK, question: Will my swap kick in automatically when I write to /dev/shm, because I have 239GB of total swap plus 128GB RAM so that’s plenty, if so.
Second question: I currently have 100G allocated to /dev/shm. Will I need in advance, to over-allocate (beyond physical RAM) my /dev/shm up to 177G (or rather 190G to be safe)? Or do I not need to set my /dev/shm size in advance to 190G because the swap will automatically take care of the usage beyond 100G I currently have allocated?
By the way my inodes is already 60 million which is plenty for my 45 million estimated requirement. THis is good. Also BTW I found that linux ext4 malfunctions at around 10.5 million files on disk, repeatably, but not deterministically. It seems to be a timeout problem due to design of something going very slowly as far as python file writing is expecting, probably a linear-time-complexity algo for table insertion of inodes of the filesystem which just takes too long for python file writing infrastructure to create any more files, despite the supposed nominal 2 billion file count limit. SSD and spinning disk behave the same here, no difference. inodes were fine and free space was fine, just always got OSErrors on true filesystems (which is why I'm going to /dev/shm) due to timeouts breaking the system.
$ uname -a Linux ga-HP-Z820 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 ga@ga-HP-Z820:/mnt/fastssd/bot_subreddit_recom/tmp$ df -i /dev/shm Filesystem Inodes IUsed IFree IUse% Mounted on tmpfs 60000000 1 59999999 1% /dev/shm ga@ga-HP-Z820:/mnt/fastssd/bot_subreddit_recom/tmp$ df -BG /dev/shm Filesystem 1G-blocks Used Available Use% Mounted on tmpfs 100G 0G 100G 0% /dev/shm ga@ga-HP-Z820:/mnt/fastssd/bot_subreddit_recom/tmp$
Here IT says
If you oversize your tmpfs instances the machine will deadlock since the OOM handler will not be able to free that memory
. So I guess I won’t do that!!! https://www.cyberciti.biz/files/linux-kernel/Documentation/filesystems/tmpfs.txt
Therefore my decision is to run an experiment to find out: Let swap take care of things. Just run it as is right now. Let my files exceed the allocated size of the /devshm, and let swap do the rest.