Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Does anyone know common reasons for such a large deficit difference in the number of files transfered when backing up my LARGE home directory using rsync on a ubuntu 10.04 LTS setup.. machine is stable and all volumes are clean ext4 no errors from fsck.ext4

Number of files: 4857743
Number of files transferred: 4203266

Thats a diff of 654,477 files!!!

I want to backup my FULL home folder to an external disk so I can fully WIPE and reformat my system and then restore my home from this rsync'd backup... but I am concerned I am missing significant data files.

I was logged in as root and used rsync to backup my /home/hholtmann/* directory to a spare backup drive in /mnt/wd750/c51/home/

Here is the command line I used as root

root@c-00000051:~# pwd
/root
root@c-00000051:~# rsync -ah --progress --stats /home/hholtmann /mnt/wd750/c51/home/ -v

Captured summary output from rsync

Number of files: 4857743
Number of files transferred: 4203266
Total file size: 487.41G bytes
Total transferred file size: 487.41G bytes
Literal data: 487.41G bytes
Matched data: 0 bytes
File list size: 102.48M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 487.75G
Total bytes received: 82.42M

Just to compare an important project sub-dir in my home after rsync:

Byte difference between a source and destination sub-dir using du

root@c-00000051:~# du -cs /home/hholtmann/proj/
18992676    /home/hholtmann/proj/
18992676    total
root@c-00000051:~# du -cs /media/wd750/c51/home/hholtmann/proj/
19006768    /mnt/wd750/c51/home/hholtmann/proj/
19006768    total

HOWEVER: NO FILE COUNT difference between the same source and destination sub-dirs

root@c-00000051:~# find /home/hholtmann/proj/ -type f -follow | wc -l
945937
root@c-00000051:~# find /mnt/wd750/c51/home/hholtmann/proj/ -type f -follow | wc -l
945937

why such unexpected results??? a file is a file.. especially in a user's home dir!!

What am I missing? or is this a sign i'm ready for management!?!

SOLUTION and ANSWERED:

The selected answer below explains for the byte count difference and my incorrect expectation of the rsync summary data..... I was just surprised by this byte difference given that both volumes are ext4 with default block sizes. I just assumed every file would take the same space in terms of du numbers.

I DID find some files that were NOT rsync'd by adding more verbose output to rsync by adding -vv to rsync and running again.

**what I saw was errors from rsync stating that it could NOT write any of my DROPBOX dir files to the destination due to the "extended attributes" on the files. rsync was skipping all my dropbox path files.

ends up my /home volume was mounted with the user_xattr ext4 mount option in the /etc/fstab file:

/dev/mapper/vg1-lv_home /home   ext4 nobarrier,noatime,user_xattr 0 2
# I HAD to add the ,user_xattr option to match my home volume
/dev/sda1           /mnt/wd750  ext4 nobarrier,noatime,user_xattr 0 2

After performing another full rsync for the 3rd time.. I decided to let a file count run all night on my full home folder and rsync'd backup:

root@c-00000051:~# find /home/hholtmann/ -type f | wc -l
4203266
root@c-00000051:~# find /mnt/wd750/c51/home/hholtmann/ -type f | wc -l
4203266

** A PERFECT MATCH OF FILES **

CONCLUSION:

** Always ensure your backup volumes are mounted with the exact same file system mount options as the source AND turn on full logging with rsync for later grep analysis to search for any errors in long file listings **

share|improve this question

migrated from serverfault.com Jan 15 at 22:05

This question came from our site for professional system and network administrators.

    
Trailing slashes in your rsync command may be a good idea here. –  ewwhite Jan 14 at 2:44
    
thanks.. but made enough mistakes in the past to learn my lesson and learn the different effect from with or without trailing slash on the source/dir/ not an issue in this case though. asking rsync to copy the full source/path/dir and create in the destination base/path/ –  Heston T. Holtmann Jan 14 at 3:05
    
Did you run rsync more then one time? Are you sure you are just looking a the fact that rsync will not re-transmit a file that is already identical between the source and destination? –  Zoredache Jan 14 at 3:44
    
Have you run a diff to see what the actual differences are? –  Michael Hampton Jan 14 at 3:52
1  
If data integrity is your goal why does the time it takes to confirm matter ? man diff is your friend in respect of how to use diff. The md5deep utility may use useful too. –  Iain Jan 14 at 7:03
show 3 more comments

2 Answers 2

up vote 2 down vote accepted

There are 2 parts to this question. First, why is there a difference between "Number of files" and "Number of files transferred". This is explained in the rsync manpage:

Number of files: is the count of all "files" (in the generic sense), which includes directories, symlinks, etc.

Number of files transferred: is the count of normal files that were updated via rsync’s delta-transfer algorithm, which does not include created dirs, symlinks, etc.

The difference here should be equal to the total amount of directories, symnlinks, other special files. Those were not "transferred" but just re-created.

Now for the second part, why is there a size difference with du. du shows the amount of disk space used by a file, not the size of the file. The same file can take up a different amount of disk space, if for example the filesystems blocksizes differ.

If you are still worried about data integrity, a easy way to be sure is to created hashes for all your files and compare:

( cd /home/hholtmann && find . -type f -exec md5sum {} \; ) > /tmp/hholtmann.md5sum
( cd /media/wd750/c51/home/ && md5sum -c /tmp/hholtmann.md5sum )
share|improve this answer
    
Thanks for taking the time to clarify the definitions for me.. that DOES explain for such a huge difference. I was able to validate my transfer in terms of total number of files between source and destination by running a file count.. see my edited question for details. –  Heston T. Holtmann Jan 15 at 3:57
add comment

Try following, this may help you,

rsync -avH --delete /home/hholtmann/ /media/wd750/c51/home
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.