2

General question

Assuming two directories with identical content are stored on different devices. Is there a way to calculate the size of the directories and always get the exact same number for both? In other words, is there such a thing as a "real size" of a directory irrespective of where it is stored?

Practical example

I transferred directories between two storage devices using rsync -ahP /dir1/ /dir2/. After the transfer finished without errors, I checked the sizes of the directories on each device using du -s --apparent-size. For some directories I got the exact same number on both devices, but not for all of them.

Specific questions

Is it possible that rsync with the chosen options didn't produce an exact copy of the directory? If yes, would there be a way to get an exact copy?

Does the used du command not give the "real size" of the directory irrespective of the storage device. If yes, would there be a way to calculate such a size?

7
  • Please define "real size" of a directory. Do you mean the size of the metadata (unit? Perhaps number of entries?), or the sum of the file sizes (not on disk) or what? The size on disk will always depend on your file system and its parameters like inode/cluster size. – Ned64 Aug 31 '19 at 11:49
  • With "real size" I mean a value that will always be the same, irrespective of the size on disk. – Matthias Aug 31 '19 at 11:53
  • Yes, but please define the size of the following data to see how you calculate it: "brw-rw---- 1 root disk 8, 0 Aug 29 09:28 /dev/sda brw-rw---- 1 root disk 8, 1 Aug 29 09:28 /dev/sda1" - how many bytes does this metadata have? (More metadata exists but is not shown.) – Ned64 Aug 31 '19 at 11:55
  • Not sure if I understood your comment. Are you saying that an objective size of a file does not exist, as the metadata will always differ on different disks? – Matthias Aug 31 '19 at 12:07
  • Yes, exactly. For example: How many bytes do you use to save a date+time? It depends on the representation (e.g. unsigned long integer as number of seconds since 01.01.1970, or 1 byte each for day, month, hour, second etc.). So, its very difficult indeed to define a definitive size. You can perhaps compare the metadata but not its size. – Ned64 Aug 31 '19 at 12:10
3

Note that du, even GNU's one with its --apparent-size option will include the apparent size (as reported by lstat()) of all types of files, including regular files, devices, symlinks, fifos, directories. GNU du like many other implementations will try to not count the same file several times (like when there are several hard links to the same file).

Here, since you're not passing the -H option to rsync, hard links won't be preserved so that exclusion of duplicates in du's account would cause a discrepancy if there were hard links in the source.

The apparent size of a file of type directory does represent the real size of its data: a list of file names alongside information on where to find them, but the format and size of that list depends on the type of file system, how it was configured, and how the directory was populated.

For device files, fifos, sockets for which rsync doesn't transfer any data, some systems (like Linux) always return 0 as the apparent size, some will return the amount of data that could be read from them (for block device files for instance, it could be the size of the corresponding storage).

So here, probably the best you can do is compute the sum of the apparent size of regular and symlink files which are the ones consistent from one system to another¹.

You could do that with GNU find with:

find . -type f,l -printf '%s\n' | awk '{s+=$0}; END{print s}'

If you find the same number on the source and destination that would be an indication that rsync may have managed to transfer all the data (the contents of regular files, and symlinks (their target path)). It may not have managed to transfer all metadata like extended attributes, ACLs (both of which you're not preserving anyway since you didn't pass the -X and -A options), file names, empty files...

As a consistent representation of the amount of directories data (assuming no encoding issue¹), you could use find . | wc -c (the sum of all file paths length + 1).

You could also re-run the same rsync command with -n (dry-run) and -v (verbose) to check if things are missing, maybe adding a --delete to also check for files that are on the destination and not the source.


¹ Strictly speaking, symlink sizes could vary if there were some transformations operated on file names like in some cases of character encoding transformations for non-ASCII characters especially if there are non-Unix or macOS file systems involved

5
  • and if the OP wanted to ignore duplicate inodes like du does, they could print the inode number as well as the size in the -printf format string: find . -type f,l -printf '%i %s\n' | awk '!seen[$1]++ { s+=$2 } ; END {print s}' – cas Sep 1 '19 at 0:55
  • @cas of course you need the device number %D in there as well as inode numbers are only unique within the device (traditionally the top level directory in each device has inode number 2). – icarus Sep 1 '19 at 5:13
  • @icarus true, i didn't think about the case of crossing filesystems. – cas Sep 1 '19 at 5:32
  • @Cas, yes though you'd only want to do it to test the effectiveness of rsync's -H option (which the OP is not using). I only brought that up as a behaviour of du that is not wanted here. I've made it more explicit now. – Stéphane Chazelas Sep 1 '19 at 6:18
  • The find command line to calculate the sum of the apparent size of regular files (I exclude symlink files for my purpose) is basically what I was looking for (now that I know that the "real size" I was asking about does not exist). – Matthias Sep 2 '19 at 18:40
3

Rsync and other tools will not copy directories exactly. They may or may not copy sparse files exactly. This is not something to be concerned about in general.

Consider the following bash example.

 mkdir -p /tmp/demo/a
 cd /tmp/demo/a
 touch {1..10000}
 ls -ld

this creates 10,000 files and lists the size of the directory holding them. On my system I get a directory of size 155648 bytes. Now remove 9,000 of them and check the size again.

 rm ????
 ls -ld

The size of the directory for me is unchanged at 155648 bytes. Now make a copy, here I am using cp but it could be rsync or cpio or anything else that copies files

 cd ..
 cp -r a b
 ls -l

For me the b directory is only 20,480 bytes, i.e. 135,168 smaller. This is because the a directory has space in it for the entry for the file 3141 which was deleted, but b doesn't have this space allocated.

1
  • That's a nice example, answering some of my questions. I accepted the other answer as it explains other aspects of my questions too. – Matthias Sep 2 '19 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.