Diagnose on ubuntu 16 OSError: [Errno 28] No space left on device: but actually plenty of space

Question

df reports no problems and plenty of space and plenty of inodes available. I can still write small new text files. Crashing python program is writing to a subdir I created in my home directory. My program is writing millions of very small files, like over 10 million, maybe much more, but well under half terabyte of bytes total I expect. THis is a (until now) lightly used conventional harddisk on a relatively new-ish workstation. Is there some way to pinpoint the problem here? Is there a quota limit in Ubuntu home directories? I only use ssh into this host and have no local kbd or GUI access, but I can do X remoting, so please limit suggestions to command lines which you can provide to me to try. thanks!

inFile: RC_2018-01-24
outDir: tmp
outputToScreenOnly: 0
Traceback (most recent call last):
  File "/mnt/fastssd/bot_subreddit_recom/write_user_docs.py", line 84, in <module>
    with open(fqfn, 'w') as f:          
OSError: [Errno 28] No space left on device: '/home/ga/reddit_data/tmp/yourstrulytony.RC_2018-01-24.doc'
^C(py36) ga@ga-HP-Z820:~/reddit_data$ df
Filesystem                   1K-blocks      Used  Available Use% Mounted on
udev                          65954704         0   65954704   0% /dev
tmpfs                         13196056      9852   13186204   1% /run
/dev/mapper/ubuntu--vg-root 1789679056 318441852 1380303752  19% /
tmpfs                         65980276         0   65980276   0% /dev/shm
tmpfs                             5120         4       5116   1% /run/lock
tmpfs                         65980276         0   65980276   0% /sys/fs/cgroup
/dev/nvme0n1                 492128608 238291700  228815144  52% /mnt/fastssd
/dev/sda2                       483946    157208     301753  35% /boot
/dev/sda1                       523248      3496     519752   1% /boot/efi
tmpfs                         13196056         4   13196052   1% /run/user/1000
(py36) ga@ga-HP-Z820:~/reddit_data$ man df
(py36) ga@ga-HP-Z820:~/reddit_data$ df /
Filesystem                   1K-blocks      Used  Available Use% Mounted on
/dev/mapper/ubuntu--vg-root 1789679056 318441852 1380303752  19% /
(py36) ga@ga-HP-Z820:~/reddit_data$ df -i /
Filesystem                     Inodes    IUsed     IFree IUse% Mounted on
/dev/mapper/ubuntu--vg-root 113647616 11444684 102202932   11% /
(py36) ga@ga-HP-Z820:~/reddit_data$ find tmp -maxdepth 1 -type f | wc -l
10603003
(py36) ga@ga-HP-Z820:~$ uname -a
Linux ga-HP-Z820 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64

Answer 1

Looks like you may be filling your ram and swap up fast enough to produce this error. Such small files take little time to propagate. Thus it is likely that they are just being created faster than they can be physically written to disk. Try adding a sleep or wait cycle between each file creation and see if that helps. Or add a read in there of some small bit of data that will cause a pause between file writes.