I've just copied a dir tree to another disk with cp -a. This tree is ~70GB big with millions of files, and diff was taking forever to finish, so I was looking for other methods to compare the dirs.

First I launched find on both the dirs and the result was the same.

After I launched du -h and one dir gave 72GB, the other 75. What can cause this difference supposing the copying process went well? Disk block size is the same, 4096. One partion is EXT3, the other EXT4, could be this?

  1. Yes, they will have different disk usages, depending on how EXT4 was tuned it could have 256 or 128 byte inodes, allocation for directories could be different, and small files can take up differing amounts of space. 3GB does sound like a lot though.

  2. My personal preferred method of comparing filesystems is:

    ( cd /src/dir; find . -type f -print0 | xargs =0 cksum ) | sort >/tmp/src.cksum
    ( cd /dst/dir; find . -type f -print0 | xargs -0 cksum ) | sort >/tmp/dst.cksum
    cksum /tmp/*.cksum

If the checksum of the two file hierarchies' filenames and checksums is the same, I feel quite confident. Faster than diff because there is no comparison, just a straight hashing of everything.

Nice part of the above method is it's easy do do via ssh between different machines as well.

  • It'll break horribly if filenames contain spaces. But I suppose since it'll break in the same way for both trees you should still end up with matching checksums. – roaima Oct 27 '18 at 21:31
  • 1
    Thanks for the reminder, I forget until I encounter a filesystem with filenames with spaces (I don't allow it on my own machines) and then reissue with -print0 and -0 Edited appropriately. – Whilom Chime Oct 27 '18 at 22:52

Probably cp -a did some sparse magic when copying the files -- or failed to do that magic; you didn't say which one is bigger, the original or the copy?

This is what the manpage of cp says:

By default, sparse SOURCE files are detected by a crude heuristic and the corresponding DEST file is made sparse as well. That is the behavior selected by --sparse=auto.

To compare the dirs and get which files are actually of different sizes, you can try comparing the output of du -S on both:

diff <(cd orig; du -S . | sort -n) <(cd copy; du -S . | sort -n)
  • the biggest dir is the original one – cdarwin Oct 27 '18 at 21:34
  • So I guess that some files that weren't sparse were detected as such and were recreated as sparse. Or files that were sparse were made even sparser, because they had long runs of zeros actually stored on disk in the original. That's not a bug -- I would've used --sparse=always in the first place. It's either that, or some file system bug ;-). I don't think that the difference between ext3 and ext4 can account for 3G of space -- but I may be wrong, of course. – mosvy Oct 27 '18 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.