How to check disk usage for folders containing reflinked files on XFS?

Question

XFS supports copy on write (CoW), so it is not entirely clear what du will say if some of the bytes are shared across files. I'd like to find a way to check how much disk space a folder uses, not counting shared bytes multiple times, i.e. the real usage on disk.

Neither xfs_estimate nor du seem to do what I need:

$ mkdir testfolder
$ cd testfolder 
$ dd if=/dev/zero of=testfile  bs=1M count=500 status=progress
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0,158889 s, 3,3 GB/s
$ cp --reflink=always testfile testfile2                      
$ xfs_estimate .                         
. will take about 1004,4 megabytes
$ du -hs .      
1000M   .

What I expect is that some tool says that this folder uses only 500MB.

df shows that free disk spaces is reduced by 500MB when using a plain cp, but not at all when doing a cp --reflink=always. So reflinking seems to work, but df is not helpful in practice, because the disk is huge and I want to check the real size of a quite small folder.

I assume this might be a valid question for BTRFS too. But in my case, I need a solution which works for XFS.

Answer 1

It feels like there should be a tool that does this by default but I don't remember if there is one.

You can query file extents using filefrag (generic, FIEMAP ioctl), or using xfs_bmap (XFS specific). That way you could choose to count shared extents (duplicates) only once (or not at all).

# filefrag -e -k testfile
Filesystem type is: 58465342
File size of testfile is 5242880 (5120 blocks of 1024 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..    5119:         96..      5215:   5120:             last,shared,eof
testfile: 1 extent found

In this example, filefrag knows and shows that the extent is shared (anywhere in the filesystem, not necessarily within that directory), xfs_bmap does not:

# xfs_bmap -l testfile
testfile:
    0: [0..10239]: 192..10431 10240 blocks

But basically that's the key ingredient you can use to script it on your own.

Show all possible shared extents:

# find . -xdev -type f -exec filefrag -e -k {} + | grep shared
   0:        0..    5119:         96..      5215:   5120:             last,shared,eof
   0:        0..    5119:       5216..     10335:   5120:             last,shared,eof
   0:        0..    5119:         96..      5215:   5120:             last,shared,eof

Shared (duplicate within directory) extents using xfs_bmap:

# find . -xdev -type f -exec xfs_bmap -l {} + | grep 'blocks$' | grep -v ': hole' | sort | uniq -d
0: [0..10239]: 192..10431 10240 blocks

Note that xfs_bmap uses 512 byte per block whereas filefrag uses 1024 bytes (with -k option) or whatever the filesystem blocksize is (like 4096 bytes).

Shared duplicate extents using filefrag:

# find . -xdev -type f -exec filefrag -ek {} + | grep shared | sort | uniq -d
   0:        0..    5119:         96..      5215:   5120:             last,shared,eof

So in this case you'd have to substract 5120 from the du -cks . result.

Answer 2

I tried to find out how much space my file really uses and how much is shared with other files. I'm not sure why "total" differs in some bytes compared to du, but finally it seems to return what I need:

file="/mnt/cache/domains/Windows 10/vdisk1-backup.img"

du -h "$file"

mapfile -t fragments < <( filefrag -ek "$file" | tail -n +4 | cut -d ":" -f 4 | grep -oP "[0-9]+" )
sum=$(IFS=+; echo "$((${fragments[*]}))")
sum=$((sum * 1024))
sum=$(echo "$sum" | numfmt --to=iec)
echo "$sum total"

mapfile -t fragments < <( filefrag -ek "$file" | tail -n +4 | grep "shared" | cut -d ":" -f 4 | grep -oP "[0-9]+" )
sum=$(IFS=+; echo "$((${fragments[*]}))")
sum=$((sum * 1024))
sum=$(echo "$sum" | numfmt --to=iec)
echo "$sum shared"

mapfile -t fragments < <( filefrag -ek "$file" | tail -n +4 | grep -v "shared" | cut -d ":" -f 4 | grep -oP "[0-9]+" )
sum=$(IFS=+; echo "$((${fragments[*]}))")
sum=$((sum * 1024))
sum=$(echo "$sum" | numfmt --to=iec)
echo "$sum unique"

Output:

15G     /mnt/cache/domains/Windows 10/vdisk1-backup.img
15G total
8.3G shared
6.8G unique

Answer 3

I made a solution for this over on the Veeam community pages:

Inherency:

Since we bill by actual disk usage for our client utilizing (Veeam) CloudConnect, the existing reports are non-functional for reporting and billing (They report pre-deduplication/reflink data). It took a while to figure out, but there is a way to calculate actual disk space used on a per directory on an immutable repository.

Solution:

Using: https://community.veeam.com/blogs-and-podcasts-57/check-reflink-and-spared-space-on-xfs-repositories-244

I have extrapolated a script that will give the disk usage of each folder (IE: client) on an immutable repo. This isn't "Data used"; that is what Veeam reports. This is "Disk Used". The actual size on disk after reflinks (duplicated data is only counted once).

A note about this script; it appears that the original blog entry is wrong on the size of a block. They attribute it to 4096... which is true... on disk... but the utility used explicitly gives the information in block sizes of 512: https://linux.die.net/man/8/xfs_bmap "units of 512-byte blocks"

To use this script, we use a cron task and pipe the output to a mail client on the repo itself (IE: script.bash 2>&1 | mail -s “Immutable storage report for $HOSTNAME” [email protected])

#!/bin/bash

for clientDir in `find /backups/disk-01/backups/ -mindepth 1 -maxdepth 1 -type d`
do
    echo $clientDir
    clientSpaceUsed=$(find $clientDir/*/* -xdev -type f -exec xfs_bmap -l {} + | awk '{ print $3 " " $4 }' | sort -k 1 | uniq | awk '{ print $2 }' | grep -Eo '[0-9]{1,7}' | paste -sd+ | bc | awk '{print $1*512/1024/1024/1024}')
    #block sizes of 512 bytes.  Divided by 1024 for KB.  Divided by 1024 for MB. Divided by 1024 for GB.
    echo "$clientSpaceUsed GB"
done

To break down how this works:

For each client directory in “/backups/disk-01/backups/”

output the directory being reported on

run xfs_bmap -l (this tells us all about the blocks in question) Take columns 3 and 4 (now becomes column 1 and 2, the rest are discarded) sort by column 1 remove duplicate rows of data (reflinks for fast cloning; keeps a single copy of the data for counting purposes) Select only column 2 (now becomes column 1)

remove anything other than numbers

Add those numbers together

multiple by block size (512)

divide by 1024 (now KB)

divide by 1024 (now MB)

divide by 1024 (now GB)

output text

Answer 4

If you just want to know the actual free space similar to df then use xfs_spaceman.

For example:

xfs_spaceman -c 'freesp -s -m4096' /path/to/xfs/mount

It can be slow to calculate this but the reported "total free blocks" is accurate with regards to reflink/deduplicated data and can be multiplied by the block size to get the actual free space in bytes.