Using rsync --link-dest for space-saving snapshots, how can I figure out how much space I actually saved? Or more general:

How to figure out how much space a directory uses considering only files that are not hardlinked elsewhere outside the directory structure? Asked differently: How much space would actually be freed after a deletion of that directory? (du -hs would lie. The space required for the hardlinks themselves may be included)

  • 2
    By default, GNU du only counts the file sizes once even if they are hard linked unless you use the -l/--count-links option. You run du on the entire tree twice, with and without that option and the difference between the sizes should be how much space you have saved over all the directories. – jw013 Oct 25 '12 at 14:17
  • 1
    See also stackoverflow.com/questions/19951883/… – Rmano Feb 20 '15 at 10:35

Assuming there aren't internal hardlinks (that is, every file with more than 1 hardlink is linked from outside the tree), you can do:

find . -links -2 -print0 | du -c --files0-from=-

EDIT And here is what I sketched in the comment, applied. Only without du; kudos to @StephaneChazelas for noticing du is not necessary. Explanation at the end.

( find . -type d -printf '%k + ' ; \
  find . \! -type d -printf '%n\t%i\t%k\n' | \
    sort | uniq -c                         | \
    awk '$1 >= $2 { print $4 " +\\" }' ; \
  echo 0 ) | bc

What we do is to create a string with the disk usage (in KB) of every relevant file, separated by plus signs. Then we feed that big addition to bc.

The first find invocation does that for directories.

The second find prints link count, inode, and disk usage. We pass that list through sort | uniq -c to get a list of (number of appearances in the tree, link count, inode, disk usage).

We pass that list through awk, and, if the first field (# of appearances) is greater than or equal the second (# of hardlinks), meaning there aren't links to this file from outside the tree, then print the fourth field (disk usage) with a plus sign and a backslash attached.

Finally we output a 0, so the formula is syntactically correct (it would en in + otherwise) and pass it to bc. Phew.

(But I would use the simpler first method, if it gives a good enough answer.)

  • Thanks, yes if that requirement is fulfilled it works. But what if it isn't? – Tobias Kienzler Oct 25 '12 at 13:45
  • That doesn't work since that fails to account for the size of directories themselves (which typically have at least 2 links, and if they hadn't, you'd have files counted twice). – Stéphane Chazelas Oct 25 '12 at 13:52
  • 1
    Then it would be necessary to use find to print a list of all files with their inodes and link count; then some combination of sort | uniq -c to get how many times each inode appears in the tree, then filter out those with link count greater than number of appearances... and then feed that list to du. But if the requirement is fulfilled, better save the effort. – angus Oct 25 '12 at 13:54
  • @StephaneChazelas It does work, but it's true that it doesn't account for the directories' own size. If only du had a -d parameter similar to ls's... – angus Oct 25 '12 at 14:05
  • Also note that on btrfs filesystems, the number of links for directories is always 1, so you'd need to add a ! -type d – Stéphane Chazelas Oct 25 '12 at 14:35

Basically, you need to get the inode numbers and number of links for all the files (non-directories), compare that number of link with the number of occurrence of each inode, and if they differ, exclude the file.

Assuming, they're all on the same filesystem, something like this should work (with GNU find):

find . -type d -printf '%k\n' -o -printf '%i %n %k\n' |
   awk '
     NF==1{t+=$0; next}
     {n1[$1]=$2; n2[$1]++; s[$1]=$3}
     END {
       for (i in n1)
         if (n1[i] == n2[i])
       print t
  • Yeah, what I said (thanks for the credit). But the extra accuracy you get by counting directories, you lose by adding inexact disk usage. – angus Oct 25 '12 at 14:12
  • @angus, what do you mean by "inexact disk usage"? – Stéphane Chazelas Oct 25 '12 at 14:20
  • Nothing, I was totally mistaken about what %k reported. That's great, du is not needed at all! I'll update my answer when I get home. Thanks! – angus Oct 25 '12 at 14:41

du actualy won't lie ;) It parses the dir(s) one gives it, counting only the first of all hardlinks pointing to the same inode it encounters.

If you ask du what it sees in one directory only, it does not care that there are other hard links pointing to the same contents:

$ du -h daily.0 && du -hc daily.1
29G /daily.0
29G /daily.1

Now give it dirs on the same row (starting with the most recent one for rsync incremental backups with --link-dest):

$ du -hc daily.0 daily.1
29G /daily.0
364M /daily.1
29G total

Or the whole backup dir:

$ du -hc --max-depth=1 /snapshots
29G /daily.0
364M /daily.1
537M /daily.2
333M /daily.3
30G total

Any file in 'daily.1' referencing an inode (aka "real" file) already referenced in 'daily.0' will not be counted.

Therefore deleting daily.1 will save 364MB on your device.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.