Since I wrote this answer, Stéphane Chazelas has convinced me that his answer was right all along. I leave my answer including code because it works well, too, and provides some pretty-printing. Its output looks like this:
total unique
--T---G---M---k---B --T---G---M---k---B
91,044,435,456 665,754,624 back-2018-03-01T06:00:01
91,160,015,360 625,541,632 back-2018-04-01T06:00:01
91,235,970,560 581,360,640 back-2018-05-01T06:00:01
91,474,846,208 897,665,536 back-2018-06-01T06:00:01
91,428,597,760 668,853,760 back-2018-07-01T06:00:01
91,602,767,360 660,594,176 back-2018-08-01T06:00:01
91,062,218,752 1,094,236,160 back-2018-09-01T06:00:01
230,810,647,552 50,314,291,712 back-2018-11-01T06:00:01
220,587,811,328 256,036,352 back-2018-11-12T06:00:01
220,605,425,664 267,876,352 back-2018-11-13T06:00:01
220,608,163,328 268,711,424 back-2018-11-14T06:00:01
220,882,714,112 272,000,000 back-2018-11-15T06:00:01
220,882,118,656 263,202,304 back-2018-11-16T06:00:01
220,882,081,792 263,165,440 back-2018-11-17T06:00:01
220,894,113,280 312,208,896 back-2018-11-18T06:00:01
Since I wasn't 100% happy with either of the two answers (as of 2018-11-18) – though I learned from both of them – I created my own tool and am publishing it here.
Similar to Stéphane Chazelas's answer, it uses find
to obtain a list of inodes and associated file / directory sizes, but doesn't rely on the "at most one link" heuristic. Instead, it creates a list of unique inodes (not files/directories!) for each input directory, filters out the inodes from the other directories, and them sums the remaining inodes' sizes. This way it accounts for possible hardlinks within each input directory. As a side effect, it disregards possible hardlinks from outside of the set of input directories.
bash-external tools that are used: find
, xargs
, mktemp
, sort
, tput
, awk
, tr
, numfmt
, touch
, cat
, comm
, rm
. I know, not exactly lightweight, but it does exactly what I want it to do. I share it here in case someone else has similar needs.
If anything can be done more efficiently or foolproof, comments are welcome! I'm anything but a bash master.
To use it, save the following code to a script file duu.sh
. A short usage instruction is contained in the first comment block.
#!/bin/bash
# duu
#
# disk usage unique to a directory within a set of directories
#
# Call with a list of directory names. If called without arguments,
# it operates on the subdirectories of the current directory.
# no arguments: call itself with subdirectories of .
if [ "$#" -eq 0 ]
then
exec find . -maxdepth 1 -type d ! -name . -printf '%P\0' | sort -z \
| xargs -r --null "$0"
exit
fi
# create temporary directory
T=`mktemp -d`
# array of directory names
dirs=("$@")
# number of directories
n="$#"
# for each directory, create list of (unique) inodes with size
for i in $(seq 1 $n)
do
echo -n "reading $i/$n: ${dirs[$i - 1]} "
find "${dirs[$i - 1]}" -printf "%i\t%b\n" | sort -u > "$T/$i"
# find %b: "The amount of disk space used for this file in 512-byte blocks."
echo -ne "\r"
tput el
done
# print header
echo " total unique"
echo "--T---G---M---k---B --T---G---M---k---B"
# for each directory
for i in $(seq 1 $n)
do
# compute and print total size
# sum block sizes and multiply by 512
awk '{s += $2} END{printf "%.0f", s * 512}' "$T/$i" \
| tr -d '\n' \
| numfmt --grouping --padding 19
echo -n " "
# compute and print unique size
# create list of (unique) inodes in the other directories
touch "$T/o$i"
for j in $(seq 1 $n)
do
if [ "$j" -ne "$i" ]
then
cat "$T/$j" >> "$T/o$i"
fi
done
sort -o "$T/o$i" -u "$T/o$i"
# create list of (unique) inodes that are in this but not in the other directories
comm -23 "$T/$i" "$T/o$i" > "$T/u$i"
# sum block sizes and multiply by 512
awk '{s += $2} END{printf "%.0f", s * 512}' "$T/u$i" \
| tr -d '\n' \
| numfmt --grouping --padding 19
# append directory name
echo " ${dirs[$i - 1]}"
done
# remove temporary files
rm -rf "$T"
dir1
saves nothing, that deletingdir2
saves nothing either, but that deleting both saves terabytes because they have large files in common that are not found elsewhere. – Stéphane Chazelas Nov 18 '18 at 17:12