How to fix intermittant "No space left on device" errors during mv when device has plenty of space?

Question

• Ubuntu 14.04 on a desktop
• Source Drive: /dev/sda1: 5TB ext4 single
  drive volume
• Target Volume: /dev/mapper/archive-lvarchive: raid6 (mdadm) 18TB volume with lvm
  partition and ext4

There are roughly 15 million files to move, and some may be duplicates (I do not want to overwrite duplicates).

Command used (from source directory) was:

ls -U |xargs -i -t mv -n {} /mnt/archive/targetDir/{}

This has been going on for a few days as expected, but I am getting the error in increasing frequency. When it started the target drive was about 70% full, now its about 90%. It used to be about 1/200 of the moves would state and error, now its about 1/5. None of the files are over 100Mb, most are around 100k

Some info:

$ df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/sdb3                      155G  5.5G  142G   4% /
none                           4.0K     0  4.0K   0% /sys/fs/cgroup
udev                           3.9G  4.0K  3.9G   1% /dev
tmpfs                          797M  2.9M  794M   1% /run
none                           5.0M  4.0K  5.0M   1% /run/lock
none                           3.9G     0  3.9G   0% /run/shm
none                           100M     0  100M   0% /run/user
/dev/sdb1                       19G   78M   18G   1% /boot
/dev/mapper/archive-lvarchive   18T   15T  1.8T  90% /mnt/archive
/dev/sda1                      4.6T  1.1T  3.3T  25% /mnt/tmp

$ df -i
Filesystem                       Inodes    IUsed     IFree IUse% Mounted on
/dev/sdb3                      10297344   222248  10075096    3% /
none                            1019711        4   1019707    1% /sys/fs/cgroup
udev                            1016768      500   1016268    1% /dev
tmpfs                           1019711     1022   1018689    1% /run
none                            1019711        5   1019706    1% /run/lock
none                            1019711        1   1019710    1% /run/shm
none                            1019711        2   1019709    1% /run/user
/dev/sdb1                       4940000      582   4939418    1% /boot
/dev/mapper/archive-lvarchive 289966080 44899541 245066539   16% /mnt/archive
/dev/sda1                     152621056  5391544 147229512    4% /mnt/tmp

Here's my output:

mv -n 747265521.pdf /mnt/archive/targetDir/747265521.pdf 
mv -n 61078318.pdf /mnt/archive/targetDir/61078318.pdf 
mv -n 709099107.pdf /mnt/archive/targetDir/709099107.pdf 
mv -n 75286077.pdf /mnt/archive/targetDir/75286077.pdf 
mv: cannot create regular file ‘/mnt/archive/targetDir/75286077.pdf’: No space left on device
mv -n 796522548.pdf /mnt/archive/targetDir/796522548.pdf 
mv: cannot create regular file ‘/mnt/archive/targetDir/796522548.pdf’: No space left on device
mv -n 685163563.pdf /mnt/archive/targetDir/685163563.pdf 
mv -n 701433025.pdf /mnt/archive/targetDir/701433025.pd

I've found LOTS of postings on this error, but the prognosis doesn't fit. Such issues as "your drive is actually full" or "you've run out of inodes" or even "your /boot volume is full". Mostly, though, they deal with 3rd party software causing an issue because of how it handles the files, and they are all constant, meaning EVERY move fails.

Thanks.

EDIT: here is a sample failed and succeeded file:

FAILED (still on source drive)

ls -lhs 702637545.pdf
16K -rw-rw-r-- 1 myUser myUser 16K Jul 24 20:52 702637545.pdf

SUCCEEDED (On target volume)

ls -lhs /mnt/archive/targetDir/704886680.pdf
104K -rw-rw-r-- 1 myUser myUser 103K Jul 25 01:22 /mnt/archive/targetDir/704886680.pdf

Also, while not all files fail, a file which fails will ALWAYS fail. If I retry it over and over it is consistent.

EDIT: Some additional commands per request by @mjturner

$ ls -ld /mnt/archive/targetDir
drwxrwxr-x 2 myUser myUser 1064583168 Aug 10 05:07 /mnt/archive/targetDir

$ tune2fs -l /dev/mapper/archive-lvarchive
tune2fs 1.42.10 (18-May-2014)
Filesystem volume name:   <none>
Last mounted on:          /mnt/archive
Filesystem UUID:          af7e7b38-f12a-498b-b127-0ccd29459376
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype needs_recovery extent 64bit flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              289966080
Block count:              4639456256
Reserved block count:     231972812
Free blocks:              1274786115
Free inodes:              256343444
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         2048
Inode blocks per group:   128
RAID stride:              128
RAID stripe width:        512
Flex block group size:    16
Filesystem created:       Thu Jun 25 12:05:12 2015
Last mount time:          Mon Aug  3 18:49:29 2015
Last write time:          Mon Aug  3 18:49:29 2015
Mount count:              8
Maximum mount count:      -1
Last checked:             Thu Jun 25 12:05:12 2015
Check interval:           0 (<none>)
Lifetime writes:          24 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      3ea3edc4-7638-45cd-8db8-36ab3669e868
Journal backup:           inode blocks

$ tune2fs -l /dev/sda1
tune2fs 1.42.10 (18-May-2014)
Filesystem volume name:   <none>
Last mounted on:          /mnt/tmp
Filesystem UUID:          10df1bea-64fc-468e-8ea0-10f3a4cb9a79
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              152621056
Block count:              1220942336
Reserved block count:     61047116
Free blocks:              367343926
Free inodes:              135953194
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      732
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         4096
Inode blocks per group:   256
Flex block group size:    16
Filesystem created:       Thu Jul 23 13:54:13 2015
Last mount time:          Tue Aug  4 04:35:06 2015
Last write time:          Tue Aug  4 04:35:06 2015
Mount count:              3
Maximum mount count:      -1
Last checked:             Thu Jul 23 13:54:13 2015
Check interval:           0 (<none>)
Lifetime writes:          150 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      a266fec5-bc86-402b-9fa0-61e2ad9b5b50
Journal backup:           inode blocks

Answer 1

Bug in the implementation of ext4 feature dir_index which you are using on your destination filesystem.

Solution : recreate filesytem without dir_index. Or disable feature using tune2fs (some caution required, see related link Novell SuSE 10/11: Disable H-Tree Indexing on an ext3 Filesystem which although relates to ext3 may need similar caution.

(get a really good backup made of the filesystem)
(unmount the filesystem)
tune2fs -O ^dir_index /dev/foo
e2fsck -fDvy /dev/foo
(mount the filesystem)

• ext4: Mysterious “No space left on device”-errors

ext4 has a feature called dir_index enabled by default, which is quite susceptible to hash-collisions.

......

ext4 has the possibility to hash the filenames of its contents. This enhances performance, but has a “small” problem: ext4 does not grow its hashtable, when it starts to fill up. Instead it returns -ENOSPC or “no space left on device”.

Answer 2

Suggestions for better-than-ext4 choices for storing masses of small files:

If you're using the filesystem as an object store, you might want to look at using a filesystem that specializes in that, possibly to the detriment of other characteristics. A quick Google search found Ceph, which appears to be open source, and can be mounted as a POSIX filesystem, but also accessed with other APIs. I don't know if it's worth using on a single host, without taking advantage of replication.

Another object-storage system is OpenStack's Swift. Its design docs say it stores each object as a separate file, with metadata in xattrs. Here's an article about it. Their deployment guide says they found XFS gave the best performance for object storage. So even though the workload isn't what XFS is best at, it was apparently better than the competitors when RackSpace was testing things. Possibly Swift favours XFS because XFS has good / fast support for extended attributes. It might be that ext3/ext4 would do ok on single disks as an object-store backend if extra metadata wasn't needed (or if it was kept inside the binary file).

Swift does the replication / load-balancing for you, and suggests that you give it filesystems made on raw disks, not RAID. It points out that its workload is essentially worst-case for RAID5 (which makes sense if we're talking about a workload with writes of small files. XFS typically doesn't quite pack them head-to-tail, so you don't get full-stripe writes, and RAID5 has to do some reads to update the parity stripe. Swift docs also talk about using 100 partitions per drive. I assume that's a Swift term, and isn't talking about making 100 different XFS filesystems on each SATA disk.

Running a separate XFS for every disk is actually a huge difference. Instead of one gigantic free-inode map, each disk will have a separate XFS with separate free-lists. Also, it avoids the RAID5 penalty for small writes.

If you already have your software built to use a filesystem directly as an object store, rather than going through something like Swift to handle the replication / load-balancing, then you can at least avoid having all your files in a single directory. (I didn't see Swift docs say how they lay out their files into multiple directories, but I'm certain they do.)

With almost any normal filesystem, it will help to used a structure like

1234/5678   # nested medium-size directories instead of
./12345678   # one giant directory

Probably about 10k entries is reasonable, so taking a well-distributed 4 characters of your object names and using them as directories is an easy solution. It doesn't have to be very well balanced. The odd 100k directory probably won't be a noticeable issue, and neither will some empty directories.

XFS is not ideal for huge masses of small files. It does what it can, but it's more optimized for streaming writes of larger files. It's very good overall for general use, though. It doesn't have ENOSPC on collisions in its directory indexing (AFAIK), and can handle having one directory with millions of entries. (But it's still better to use at least a one-level tree.)

Dave Chinner had some comments on XFS performance with huge numbers of inodes allocated, leading to slow-ish touch performance. Finding a free inode to allocate starts taking more CPU time, as the free inode bitmap gets fragmented. Note that this is not an issue of one-big-directory vs. multiple directories, but rather an issue of many used inodes over the whole filesystem. Splitting your files into multiple directories helps with some problems, like the one that ext4 choked on in the OP, but not the whole-disk problem of keeping track of free space. Swift's separate-filesystem-per-disk helps with this, compared to on giant XFS on a RAID5.

I don't know if btrfs is good at this, but I think it may be. I think Facebook employs its lead developer for a reason. :P Some benchmarks I've seen, of stuff like untarring a Linux kernel source, show btrfs does well.

I know reiserfs was optimized for this case, but it's barely, if at all, maintained anymore. I really can't recommend going with reiser4. It might be interesting to experiment, though. But it's by far the least future-proof choice. I've also seen reports of performance degradation on aged reiserFS, and there's no good defrag tool. (google filesystem millions of small files, and look at some of the existing stackexchange answers.)

I'm probably missing something, so final recommendation: ask about this on serverfault! If I had to pick something right now, I'd say give BTRFS a try, but make sure you have backups. (esp. if you use BTRFS's build-in multiple disk redundancy, instead of running it on top of RAID. The performance advantages could be big, since small files are bad news for RAID5, unless it's a read-mostly workload.)

Answer 3

For this issue below is what i did to fix( you may need sudo access for below steps):

1. Used space of Inodes was 100% which can be retrieved using below command

   df -i /

Filesystem Inodes IUsed IFree IUse% Mounted on

/dev/xvda1            524288   524288  o     100% /

2. Need to free up the iNoted, hence need to find the files which has here number of i nodes using below command:

Try to find if this is an inodes problem with:

df -ih

Try to find root folders with large inodes count:

for i in /*; do echo $i; find $i |wc -l; done

Try to find specific folders:

for i in /src/*; do echo $i; find $i |wc -l; done

3. now we have zeroed down to the folder with large number of files in it. Run the below commands one after the other to avoid any error (In my case the actual folder was /var/spool/clientmqueue):

find /var/spool/clientmqueue/ -type f -mtime +1050 -exec rm -f {} +

find /var/spool/clientmqueue/ -type f -mtime +350 -exec rm -f {} +

find /var/spool/clientmqueue/ -type f -mtime +150 -exec rm -f {} +

find /var/spool/clientmqueue/ -type f -mtime +50 -exec rm -f {} +