Reasons for rsync NOT transferring all files?

Question

Does anyone know common reasons for such a large deficit difference in the number of files transferred when backing up my LARGE home directory using rsync on a Ubuntu 10.04 LTS setup? The machine is stable and all volumes are clean ext4 -- no errors from fsck.ext4.

Number of files: 4857743
Number of files transferred: 4203266

That's a difference of 654,477 files!!!

I want to backup my FULL home folder to an external disk so I can fully WIPE and reformat my system and then restore my home from this rsync'd backup, but I am concerned I am missing significant data files.

I was logged in as root and used rsync to backup my /home/hholtmann/* directory to a spare backup drive in /mnt/wd750/c51/home/

Here is the command line I used as root

root@c-00000051:~# pwd
/root
root@c-00000051:~# rsync -ah --progress --stats /home/hholtmann /mnt/wd750/c51/home/ -v

Captured summary output from rsync

Number of files: 4857743
Number of files transferred: 4203266
Total file size: 487.41G bytes
Total transferred file size: 487.41G bytes
Literal data: 487.41G bytes
Matched data: 0 bytes
File list size: 102.48M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 487.75G
Total bytes received: 82.42M

Just to compare an important project sub-dir in my home after rsync:

Byte difference between a source and destination sub-dir using du

root@c-00000051:~# du -cs /home/hholtmann/proj/
18992676    /home/hholtmann/proj/
18992676    total
root@c-00000051:~# du -cs /media/wd750/c51/home/hholtmann/proj/
19006768    /mnt/wd750/c51/home/hholtmann/proj/
19006768    total

HOWEVER: NO FILE COUNT difference between the same source and destination sub-dirs

root@c-00000051:~# find /home/hholtmann/proj/ -type f -follow | wc -l
945937
root@c-00000051:~# find /mnt/wd750/c51/home/hholtmann/proj/ -type f -follow | wc -l
945937

why such unexpected results? A file is a file... especially in a user's home dir!

What am I missing? Or is this a sign I'm ready for management!?!

SOLUTION and ANSWERED:

The selected answer below explains for the byte count difference and my incorrect expectation of the rsync summary data. I was just surprised by this byte difference given that both volumes are ext4 with default block sizes. I just assumed every file would take the same space in terms of du numbers.

I DID find some files that were NOT rsync'd by adding more verbose output to rsync by adding -vv to rsync and running again.

What I saw was errors from rsync stating that it could NOT write any of my DROPBOX dir files to the destination due to the "extended attributes" on the files. rsync was skipping all my dropbox path files.

Ends up my /home volume was mounted with the user_xattr ext4 mount option in the /etc/fstab file:

/dev/mapper/vg1-lv_home /home   ext4 nobarrier,noatime,user_xattr 0 2
# I HAD to add the ,user_xattr option to match my home volume
/dev/sda1           /mnt/wd750  ext4 nobarrier,noatime,user_xattr 0 2

After performing another full rsync for the 3rd time, I decided to let a file count run all night on my full home folder and rsync'd backup:

root@c-00000051:~# find /home/hholtmann/ -type f | wc -l
4203266
root@c-00000051:~# find /mnt/wd750/c51/home/hholtmann/ -type f | wc -l
4203266

** A PERFECT MATCH OF FILES **

CONCLUSION:

** Always ensure your backup volumes are mounted with the exact same file system mount options as the source AND turn on full logging with rsync for later grep analysis to search for any errors in long file listings! **

Answer 1

To all other poor lost souls working from vacation in the dead of night,

--checksum makes rsync actually check if there are changes in the files, otherwise it checks timestamps and file sizes and calls it a day,

this is sufficient in 99.9% of the cases and lets you burn in hell for the rest of the 0.01% until you figure this out

Answer 2

There are 2 parts to this question. First, why is there a difference between "Number of files" and "Number of files transferred". This is explained in the rsync manpage:

Number of files: is the count of all "files" (in the generic sense), which includes directories, symlinks, etc.

Number of files transferred: is the count of normal files that were updated via rsync’s delta-transfer algorithm, which does not include created dirs, symlinks, etc.

The difference here should be equal to the total amount of directories, symnlinks, other special files. Those were not "transferred" but just re-created.

Now for the second part, why is there a size difference with du. du shows the amount of disk space used by a file, not the size of the file. The same file can take up a different amount of disk space, if for example the filesystems blocksizes differ.

If you are still worried about data integrity, a easy way to be sure is to created hashes for all your files and compare:

( cd /home/hholtmann && find . -type f -exec md5sum {} \; ) > /tmp/hholtmann.md5sum
( cd /media/wd750/c51/home/ && md5sum -c /tmp/hholtmann.md5sum )

Answer 3

I might as well add something I learned.

I was using the command rsync /path/source/* /path/to/destination/* (notice the globbing). It was awkward because 90% of my files had transferred with a few exceptions (even being in the same folder as some that did transfer). After removing the * from source and destination, they all transferred. ¯\_(ツ)_/¯

Answer 4

Try following, this may help you,

rsync -avH --delete /home/hholtmann/ /media/wd750/c51/home

Answer 5

If you are attempting to copy your entire /var folder to an external drive using rsync, but you run into issues with rsync like this:

sent 169,748,362,821 bytes  received 3,982,112 bytes  16,898,347.01 bytes/sec
total size is 193,085,110,161  speedup is 1.14
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0] 

the best course of action is to NOT copy the entire /var folder, but instead only copy the /var/www folder. Using rsync with /var/lib is usually where the files fail to transfer, to circumvent this you can use the mysql dump command which will backup all databases to one file. Here is an example of my backup process for my linux webserver

Backup protocol:
sudo fdisk -l
mount /dev/sdb1 /mnt
mysqldump -u root -p --all-databases > alldbs.sql
mv alldbs.sql /mnt/backup
rsync -av /var /mnt/backup (will likely fail to copy entire folder)
rsync -av /var/www /mnt/backup 
rsync -av /home /mnt/backup
rsync -av /etc /mnt/backup
umount -l /dev/sdb1 

Above command descriptions, in order:
lists all disks on system
mounts external drive (sdb1 - [partition of sdb] could be different)
dumps all mysql databases to a file called alldbs.sql
moves all databases file onto backup drive
copies almost the entire var directory into the drive backup folder
copies entire var/www web directory into the drive backup folder
copies entire home directory into the drive backup folder
copies entire etc directory into the drive backup folder
unmounts the external drive (sdb1)  [using -l because the filesystem is still accessing the /mnt folder, but everything is already copied]

rsync flags options explained:

-a: archive mode: preserves file permissions, recursively copies, copies symlinks, preserves modification times, preserves groups, preserves file ownership, preserves device files
-v: verbose mode: displays which files are being copied 

If you just wish to have the data transfered as ASCII rather than binary, you can use the rsync flag option -h, which stands for "human readable." This is not recommended at all...

Now lets talk filesystem format of the exteral drive. Preferably, you want to use ext4 to preserve all permissions on the linux system, but you can alternatively use NTFS if you plan on modifying the permissions later:

Safety aside (which as other users have established should unlikely be an issue), you must keep in mind that the permission system on NTFS and ext4 is different and cannot be "carried over" from one file system to another without loss of information.

For instance, if you mark a script as "executable" on your main ext4 drive and then copy it to the NTFS backup and back, it will no longer be marked as such. Similarly, files marked as "read only" will become accessible to everyone if you mount the NTFS partition with the standard 644 file permissions (i.e. everyone can read the file, but only the owner can write into it).

All this means that while restoring media like videos or photos from a backup should be absolutely fine, restoring an app may require you to tweak permissions and ownership for its executables. Likewise, if you restore a file that was meant to be read only by you (400), everyone will be able to read it unless you remember to re-set its permissions.

One workaround to the problem above might be be putting all your backed-up data into a UNIX-native container such as tar that will preserve such information. Doing incremental backups with tar is difficult, however, and you may end up with the same file taking hard drive space in multiple tarred snapshots.

Considering the above, I would suggest formatting the drive to your system-native file system, in your case ext4, backing up your data with rsync, and reformatting it back to NTFS if you need to repurpose that drive later on for Windows.

If you feel a little bit adventurous, you may want to take a look at the Btr(fs) file system due to its built-in copy-on-write optimization and snapshots that may come in handy for backups. Last time I checked it had issues, but also looked very promising.