0

I've just copied across some data from one raid partition to another. When I look at the output of df:

Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/md127     1921673480 1652707240  171280736  91% /home/chris/mount/raid18t
/dev/md3       2881491232 1652719488 1082329968  61% /home/chris/mount/raid3t

The Used values are not identical (1652707240 vs 1652719488). However, if I do a du -s * in each of the directories though - I get the same values/output...

Why is there a difference in the Used values?

Both filesystems are ext4.

5
  • 3
    Do you have many directories in that data set? Have files been added and deleted over some time? If so, the directory nodes themselves would take somewhat more space than newly created directories. This depends on the filesystem though.
    – Kusalananda
    May 21, 2020 at 16:15
  • Not sure what you mean by "data set"?. The files should be identical - I did a cp -a .... May 21, 2020 at 16:22
  • 1
  • Showing a list of files, file systems on both devices and du on both would be useful. It's possible that the FS under /dev/md3 takes up more metadata (or has more space reserved for it) since it has to handle more 1k blocks than that of /dev/md127.
    – Pedro
    May 21, 2020 at 16:36
  • @ChrisStryczynski What are the filesystems of /dev/md127 and /dev/md3 ?
    – binarysta
    May 21, 2020 at 16:41

1 Answer 1

2

The reason is, df reports the size on disk whereas du reports on actual file size. Even in the case of files being exactly the same, does not imply that the underlying file system blocks are guaranteed to be identical.

Consider a block size of 8k on Disk-1 and a block size of 4k on Disk-2.
File foo has an actual size of 11k.

Disk-1/foo will take 2 blocks, which uses 28k=16k.
Disk-2/foo will take 3 blocks, which uses 3
4k=12k.

EDIT
Just for fun, pick a directory with lots of small files in it. run ls -l and total the file sizes. This is the actual size of the file. On my flavor of Linux, the file size is in column 5. So I'd do something like this:
ls -l | awk '{ttl=ttl+$5} END {print "Total file size " ttl}'
This will total the file sizes in the directory.
Next, list the number of sectors on the disk, with ls -ls and total the sectors:
ls -ls | awk '{ttl=ttl+$1} END {print "Total sector count " ttl}'

On my system, I checked my boot directory:
ls -lsR /boot | awk '{if (($1+0) >0) tt_sec=tt_sec+$1; tt_byt=tt_byt+$6} END {print "Sectors " tt_sec " Bytes " tt_byt}'
Sectors 151,352 Bytes 153,894,246
du -xs /boot
151360
df /boot
1-K blocks: 499656
Used: 151756

Notice how all 3 numbers are different? ls shows the size in bytes, by default.
df unless otherwise configured, reports 1K blocks
My disk is configured with 512 byte sectors (I got that from fdisk)
If I add hidden files & Directories into my ls command:
ls -lRsa (…) Sectors 151,504 Bytes 154,045,696
Closer, but still off.
Let me show you, now, what is happening:
-copied for convenience-
ls -lsRa /boot | awk '{if (($1+0) >0) tt_sec=tt_sec+$1; tt_byt=tt_byt+$6} END {print "Sectors " tt_sec " Bytes " tt_byt}'
Sectors 151,504 Bytes 154,045,696
df /boot
1-K blocks: 499656
Used: 151756
du -xs /boot
151360

echo "Hi" > junk ls -ls junk
4 -rw-r--r-- ScottieH user 3 junk
Notice that this 3-byte file is using 4 sectors!

-recompute sizes- ls -lsRa /boot | awk '{if (($1+0) >0) tt_sec=tt_sec+$1; tt_byt=tt_byt+$6} END {print "Sectors " tt_sec " Bytes " tt_byt}'
Sectors 151,508 Bytes 154,045,972 <-How much did this go up?
du -xs /boot
151364
df /boot
1-K blocks: 499656
Used: 151760

** Do you see how these numbers will NEVER add up?**

4
  • It's clear to me that it measures something else, but my question is why does it differ? Should it not be identical? May 21, 2020 at 16:58
  • Block size seems to be identical according to hdparm -I /dev/... | grep Physical May 21, 2020 at 16:58
  • I do not know what hdparm does. I'll have to look at it. Really, you are looking for the sector size on the disk. The difference in the sizes is, in summary, the difference in the actual file size, and the size on disk.
    – Scottie H
    May 22, 2020 at 0:08
  • I ran hdparm on my /boot drive. I get different answers between hdparm, fdisk and cfdisk. Similar, but different..
    – Scottie H
    May 22, 2020 at 1:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .