27

I'm migrating a server from an Ubuntu Server 18.02 instance ("saturn") to a newly-built Debian Buster 10 system ("enceladus"). I have copied a complete filesystem across the network using

sudo rsync --progress -au --delete --rsync-path="sudo rsync" /u/ henry@enceladus:/u

I check the number of directories and the number of files on the sending and receiving side: the counts are identical. I have an RYO Perl program which traverses the file tree and compares each file in one tree with its counterpart in the other: it finds no differences in 52,190 files. Both filesystems are EXT4; both have 512-byte blocks logical, 4096 physical.

Yet the receiving filesystem is 103,226,592,508 bytes and the sending one only 62,681,486,428. If the received filesystem were a little smaller I could understand it, because of unreclaimed blocks; but it's the other way round, and the difference is two thirds the original!

How can this be? Should I worry about it, as being evidence of some malfunction?

  • 6
    What is it actually measuring? The most obvious thing would be that some files on the sending end are "sparse", i.e. they have regions where all the data is NUL bytes and these are just noted as such, rather than stored as disk blocks full of zeros. The receiving file could actually have disk blocks allocated. – icarus Jan 4 at 1:42
  • I have always used -axHAWXS for file system cloning – Richie Frame Jan 4 at 16:40
  • @RichieFrame Could you please explain what the relevant difference is to the command OP used since otherwise your comment isn't very useful. – Martijn Heemels Jan 6 at 8:28
  • @MartijnHeemels it is useful if you are comparing those to the options used when looking at the options list. Compared to -au, it adds preservation of hardlinks, ACLs, and extended attributes, it disables the delta transfer, handles sparse files efficiently, and prevents recursion across a filesystem boundry so mount point contents are not copied, as those should be handled as their own filesystem – Richie Frame Jan 6 at 8:52
79

I can think of two things offhand:

  • you didn't use -H, so hardlinks are lost.
  • you didn't use -S, so sparse files may have been expanded
  • 14
    Thank you for very helpful suggestions. There are no sparse files, but there are significant numbers of hard links, with large-ish images at the end of them. I'm re-doing the copy (from scratch) with -H and will post the result. – Henry Law Jan 3 at 18:38
  • 37
    ... and that gave the expected result, to within a few K. Thank you very much. – Henry Law Jan 3 at 19:03
  • 7
    Yeah, it's kind of surprising (in the UI sense) that -a does not include these options. – mattdm Jan 3 at 20:54
  • 13
    @mattdm the -a flag omits -H (hard links) because handling this requires holding the entire tree of linked files in memory so that matching inodes can be identified. It omits both -H and -S because not all filesystems can support these features – roaima Jan 3 at 21:26
  • 12
    Oh, I know why it doesn't include them. It's just kind of a trap, as evidenced here. – mattdm Jan 3 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.