4

I have copied a folder using rsync including symlinks, hard links, permissions, deleting files on destination et cetera. They should be pretty identical.

One folder is on a USB drive and the other on a local disk.

If I run: du -bls on both folders, the size comes up as slightly different.

My du supports --apparent-size and it is applied by -s and -l should count the content of the hard links.

How can this difference be explained and how do I get the actual total?

Both file systems are ext4, the only difference is that the USB drive is encrypted.

EDIT:

I digged down to find the folders that were actually different, I found one and the content is not special (no block device, no pipes, no hard or symlinks, no zero bytes files), the peculiarity may be having several small files within it. The difference is 872830 vs 881022 of this particular folder.

I also ran du -blsc in both folders and the result is the same in this case.

Some extra details on the commands I used:

$ du -Pbsl $LOCALDIR $USBDIR | cut -f1
872830
881022

$ du -Pbslc $LOCALDIR/*
[...]
868734  total

$ du -Pbslc $USBDIR/*
[...]
868734  total

$ ls -la $USBDIR | wc
    158    1415    9123
$ ls -la $LOCALDIR | wc
    158    1415    9123

$ diff -sqr --no-dereference $LOCALDIR $USBDIR | grep -v identical
[No output and all identical if I remove the grep]
  • 5
    Define "pretty identical" and "slightly different". – Kusalananda May 11 at 14:47
  • How much of a difference? – Atul May 11 at 14:55
  • @Kusalananda pretty identical = identical content, user, permissions, timestamp for each file/folder, slightly different = a small amount of bytes in difference – Stefano d'Antonio May 11 at 14:56
  • @Atul 836034841990 vs 836037115270 (the content is then roughly 800GB) – Stefano d'Antonio May 11 at 14:57
  • That's about 2 MB. Are you able to run a md5sum over the files and then verify that against the other set? I wonder if you have a lot of directories that could account for the difference (some filesystems don't truncate the directories when you delete entries)? – Kusalananda May 11 at 15:00
11

Since you have copied the files using rsync and then compared the two sets of files using diff, and since diff reports no difference, the two sets of files are identical.

The size difference can then probably be explained by the sizes of the actual directory nodes within the two directory structures. On some filesystems, the directory is not truncated if a file or subdirectory is deleted, leaving a directory node that is slightly larger than what's actually needed.

If you have, at some point, kept many files that were later deleted, this might have left large directory nodes.

Example:

$ mkdir dir
$ ls -ld dir
drwxr-xr-x  2 kk  wheel  512 May 11 17:09 dir
$ touch dir/file-{1..1000}
$ ls -ld dir
drwxr-xr-x  2 kk  wheel  20480 May 11 17:09 dir
$ rm dir/*
$ ls -ld dir
drwxr-xr-x  2 kk  wheel  20480 May 11 17:09 dir
$ du -h .
20.0K   ./dir
42.0K   .
$ ls -R
dir

./dir:

Notice how, even though I deleted the 1000 files I created, the dir directory still uses 20 KB.

  • That is quite interesting, I also used du -bs and I was able to reproduce what you described. Would be interesting to know what ext4 does. – Stefano d'Antonio May 11 at 15:17
  • @Stefanod'Antonio I believe that ext4 behaves the same. My tests were on an OpenBSD system using its native FFS filesystem. – Kusalananda May 11 at 15:20
  • @Kusalanada what I meant is how this works behind the scenes: what's the threshold and why it does that. – Stefano d'Antonio May 11 at 15:21
  • 1
    @Stefanod'Antonio Possibly to reduce filesystem fragmentation. There is no threshold. The directory node is simply never truncated. – Kusalananda May 11 at 15:22
  • 1
    @danieldeveloper001 I'm not a Linux user and don't know if there's some specific tool for doing this on ext4 filesystems, but the portable way would be to move the contents of a directory to a new directory and then rmdir the original directory. Or, for a whole hierarchy, use rsync to copy it, then delete the original (as the user in the question actually did). – Kusalananda May 11 at 15:31
1

Have you checked the filesystem block size? Even though both devices use the same filesystem, it is a possibility that the block sizes are different and this could explain the "slightly different" sizes.

When, for instance, storing a bunch of 1KiB files in a device with filesystem set to use 8KiB block size, there will be a waste of 7KiB per used block. The actual size that your files are taking from your disk, is the size of the used blocks, not the size of the files itself in this case (unless there is some kind of tool to store multiple files per block). Try checking your different devices block size with the command below.

# blockdev --getbsz <DEVICE>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.