1

I have a script I'm testing on some locations on my volumes, and part of it is the calculation of disk usage & total data size. It was all fine, and the values I was getting always matched what macOS itself was reporting in the mdls output and in Finder Info.

However, when running the script on /usr/local/Cellar – the Homebrew installation directory – the output was weird. Disk usage was at 3,234,828,288 bytes, while the data size was actually higher, namely at 3,254,656,870 bytes. Same result when running the script as root or with GNU-du. Shouldn't the disk usage always be higher than the data size? (Unless you're using HFS+ compression, of course, but that doesn't seem to be the case here: there are no "compressed" file flags in the ls output.)

Here are the relevant parts of the script, first disk usage, then data total:

FILEPATH="/usr/local/Cellar"
DISK_USAGE=$(/usr/bin/du -k -d 0 "$FILEPATH" | /usr/bin/head -n 1 | /usr/bin/awk '{print $1}')
DU_SIZE=$(echo "$DISK_USAGE * 1024" | /usr/bin/bc -l)
SEDPATH=$(echo "$FILEPATH" | /usr/bin/awk '{gsub("/","\\/");print}')
LISTING=$(ls -RAlO@ "$FILEPATH" | /usr/bin/sed -e '/^$/d' -e '/^'"$SEDPATH"'/d' | /usr/bin/awk 'NF>2')
DATASIZE=$(echo "$LISTING" | /usr/bin/grep -v '^d' | /usr/bin/awk '{total += $6} END {printf "%.0f", total}')
echo "du: $DU_SIZE"
echo "data: $DATASIZE"

Probable solution: macOS & HFS+ are the problem, as always! They call it "Apple Core Rot", I believe. I ran the du command again, together with the script to calculate the total data size, and also added some commands to calculate the disk usage "manually" from the block count in the ls/stat output, and lo and behold, calculated disk usage is in fact larger than the total data size, as it should be without FS compression (sparse files etc.), and now (all of a sudden!) the du command also spits out the correct result. Why I can only guess, but it's probably because HFS+ hadn't gotten around to calculating it in the first place, so when du asked the system "what's the disk usage", it only got a partial result.

FILEPATH="/usr/local/bin"                                                                                                     [21:38:27]
SEDPATH=$(echo "$FILEPATH" | /usr/bin/awk '{gsub("/","\\/");print}')
MPOINT=$(/bin/df "$FILEPATH" | /usr/bin/tail -1 | /usr/bin/awk '{for(i=9; i<=NF; i++) printf "%s",$i (i==NF?ORS:OFS)}')
CLUSTERSIZE=$(/usr/sbin/diskutil info "$MPOINT" | /usr/bin/awk '/Device Block Size/{print $4}')
TOTAL_LIST=$(ls -ReAlOs@ "$FILEPATH")
SWITCH_LIST=$(echo "$TOTAL_LIST" | /usr/bin/sed -e '/^$/d' -e '/^'"$SEDPATH"'/d' | /usr/bin/awk 'NF>=11' | /usr/bin/awk '{print $2,$7,$1}')
BLOCKSONDISK=$(echo "$SWITCH_LIST" | /usr/bin/grep -v '^d' | /usr/bin/awk '{total += $3} END {printf "%.0f", total}')
SIZEONDISK=$(echo "$BLOCKSONDISK * $CLUSTERSIZE" | /usr/bin/bc -l)
echo "od: $SIZEONDISK B"
DISK_USAGE=$(/usr/bin/du -k -d 0 "$FILEPATH" | /usr/bin/head -n 1 | /usr/bin/awk '{print $1}')
DU_SIZE=$(echo "$DISK_USAGE * 1024" | /usr/bin/bc -l)
echo "du: $DU_SIZE B"
DATASIZE=$(echo "$SWITCH_LIST" | /usr/bin/grep -v '^d' | /usr/bin/awk '{total += $2} END {printf "%.0f", total}')
echo "ds: $DATASIZE B"

Output:

physical (stat): 36343808 B
physical   (du): 36343808 B
datasize (stat): 32584254 B
  • "Shouldn't the disk usage always be higher than the data size?" Not necessarily — sparse files can take less space on disk than their size – Fox Mar 25 '17 at 0:52
  • Ah, good to know. I just searched the dir with the GNU solution gfind . -type f ! -size 0 -printf '%S:%p\0' | /usr/local/bin/sed -zn 's/^0[^:]*://p' | gtr '\0' '\n' … nothing there. – JayB Mar 25 '17 at 7:47
  • See here — does that offer any insight? (The claim is the du does not count space occupied by inodes) – Fox Mar 25 '17 at 14:01
  • I think I'll try to calculate the disk usage manually first, namely by adding the s option to ls, i.e. ls -RealOs@ … column 1 will then have the number of blocks used by a file or symlink etc., and you can just total it with awk, then multiply by the FS block size, the device block (= cluster) size, not the allocation block size, which would be MPOINT=$(df "$FILEPATH" | tail -1 | awk '{for(i=9; i<=NF; i++) printf "%s",$i (i==NF?ORS:OFS)}') && diskutil info "$MPOINT" | awk '/Device Block Size/{print $4}' … and see if this gets me anywhere in comparison to the du results. – JayB Mar 25 '17 at 20:15
  • edited the original post; seems to be a macOS/HFS+ problem. Jesus! – JayB Mar 25 '17 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.