xfs disk usage is 15-30% bigger than the files

Question

xfs seems to use 15-30% more disk space than the files require:

# du
0       .
# dd if=/dev/zero of=test bs=1M count=5k
5120+0 records in
5120+0 records out
5368709120 bytes (5,4 GB) copied, 10,527 s, 510 MB/s
# ls -l
total 8388608
-rw-r--r-- 1 root root 5368709120 Oct 19 16:04 test
# du
8388608 .

It seems to do that to a certain degree for the whole file system. This filesystem reports disk full when adding files less than 1% of the file system's size:

# df
/dev/sdb6  40957913088 35624042728 5333870360  87% /xfs-export

Is it caused by some fancy option when doing mkxfs (similar to the reserved 5% in ext2)?

# uname -a
Linux server 2.6.39-bpo.2-amd64 #1 SMP Tue Jul 26 10:35:23 UTC 2011 x86_64 GNU/Linux

Answer 1

I strongly suspect you're seeing such strange numbers because you have the environmental variable BLOCKSIZE set to 640. BLOCKSIZE affects what ls, du, and df print. Set BLOCKSIZE to "1024" or "1k" and you should see the output you expect.

Answer 2

It has been a long time since I could reproduce this problem. I think I got it now:

$ ls -l
total 0
$ dd if=/dev/md0 >> a bs=10000k &
$ ls -l
total 9111552
-rw-r--r-- 1 rt rt 5816320000 Jan  7 23:31 a

It stays at this until dd finishes (or is killed) - even if dd is suspended. After that things look normal:

$ kill %1; ls -l
total 20120000
-rw-r--r-- 1 rt rt 20602880000 Jan  7 23:33 a

So it simply looks as if xfs is reserving space for the file to expand on.

The dd in the question finishes, but there might have been some other files open that could justify the reservation.

Answer 3

XFS has a feature called Dynamic Speculative EOF Preallocation. It could allocate more space in expect of more bytes being written to reduce file fragmentation. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=055388a3188f56676c21e92962fc366ac8b5cb72)