Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I used rsync to copy a large number of files, but my OS (Ubuntu) restarted unexpectedly.

After reboot, I ran rsync again, but from the output on the terminal, I found that rsync still copied those already copied before. But I heard that rsync is able to find differences between source and destination, and therefore to just copy the differences. So I wonder in my case if rsync can resume what was left last time?

share|improve this question
    
Yes, rsync won't copy again files that it's already copied. There are a few edge cases where its detection can fail. Did it copy all the already-copied files? What options did you use? What were the source and target filesystems? If you run rsync again after it's copied everything, does it copy again? –  Gilles Sep 16 '12 at 1:56
    
@Gilles: Thanks! (1) I think I saw rsync copied the same files again from its output on the terminal. (2) Options are same as in my other post, i.e. sudo rsync -azvv /home/path/folder1/ /home/path/folder2. (3) Source and target are both NTFS, buy source is an external HDD, and target is an internal HDD. (3) It is now running and hasn't finished yet. –  Tim Sep 16 '12 at 2:30
    
There is also the --partial flag to resume partially transferred files (useful for large files) –  jwbensley Sep 16 '12 at 16:15
    
@Gilles: What are some "edge cases where its detection can fail"? –  Tim Sep 19 '12 at 5:20
1  
@Tim Off the top of my head, there's at least clock skew, and differences in time resolution (a common issue with FAT filesystems which store times in 2-second increments, the --modify-window option helps with that). –  Gilles Sep 19 '12 at 9:25

3 Answers 3

You may want to add the -P option to your command.

From the man page:

  -P     The -P option is equivalent to --partial --progress.   Its  pur-
         pose  is to make it much easier to specify these two options for
         a long transfer that may be interrupted.

So instead of:

sudo rsync -azvv /home/path/folder1/ /home/path/folder2

Do:

sudo rsync -azvvP /home/path/folder1/ /home/path/folder2

Of course, if you don't want the progress updates, you can just use --partial, i.e.:

sudo rsync --partial -azvv /home/path/folder1/ /home/path/folder2
share|improve this answer

First of all, regarding the "resume" part of your question, --partial just tells the receiving end to keep partially transferred files if the sending end disappears. Such files will exist as hidden files in their target folders (e.g. .TheFileYouAreSending.lRWzDC), or a specifically chosen folder if you also set the --partial-dir switch.

--partial doesn't facilitate resuming a failed or cancelled transfer. Notably, a cancelled transfer (aborted with Ctrl+C) will stop the operation gracefully and the receiving end will actually delete the unfinished partial file from the target. The point of partial files is to avoid polluting the target with files that look ok but aren't (yet).

So, first of all, if you're moving large files and you want the option to cancel the rsync operation at will and resume later where you left off, you need to use the --append switch. This will append data to smaller files that exist in the target with the same name rather than replace them with the source's version.

--append isn't dangerous: It will always read and compare the data on both ends and not just assume they're equal. It does this using checksums, so it's easy on the network, but it does require reading the shared amount of data on both ends of the wire before it can actually resume the transfer by appending to the target.

Note that if you want to be able to Ctrl+C a running rsync operation and resume it later, you should not use --partial as well, because as mentioned, Ctrl+C is a graceful way of cancelling the operation and it will cause rsync to delete the partial file on the target.

Second of all, you said that you "heard that rsync is able to find differences between source and destination, and therefore to just copy the differences."

That's correct, and it's called delta transfer, but it's a different thing. To enable this, you add the -c, or --checksum switch. Once this switch is used, rsync will examine files that exist on both ends of the wire. It does this in chunks, compares the checksums on both ends, and if they differ, it transfers just the differing parts of the file.

This is a bit computationally heavy on both ends initially, but can be extremely efficient to reduce network load if for example you're frequently backing up very large files that often contain minor changes. Examples that come to mind are virtual hard drive image files used in virtual machines or iSCSI targets.

It is notable that if you use --checksum to transfer a batch of files that are completely new to the target system, rsync will still calculate their checksums on the source system before transferring them. Why I do not know :)

So, in short:

If you're often using rsync to just "move stuff from A to B" and want the option to cancel that operation and later resume it, don't use --checksum, but do use --append.

If you're using rsync to back up stuff often, using --checksum can be very benificial, --append probably not, unless you're in the habit of sending large files that continously grow in size but are never modified once written (and --checksum would work just as well as --append here).

When using --append, rsync will behave just like it always does on all files that are the same size. If they differ in modification or other timestamps, it will overwrite the target with the source without scrutinizing those files further. --checksum will always compare the contents (checksums) of every file, regardless of size or attributes.

share|improve this answer
    
That should be marked as the best answer! –  Christian Nov 24 at 22:10

i think you are forcibly calling the rsync and hence all data is getting downloaded when you recall it again. use --progress option to copy only those files which are not copied and --delete option to delete any files if already copied and now it does not exist in source folder...

rsync -avz --progress --delete -e  /home/path/folder1/ /home/path/folder2

if you are using ssh to login to other system and copy the files,

rsync -avz --progress --delete -e "ssh -o UserKnownHostsFile=/dev/null -o \
StrictHostKeyChecking=no" /home/path/folder1/ /home/path/folder2

let me know if there is any mistake in my understanding of this concept...

share|improve this answer
1  
Can you please edit your answer and explain what your special ssh call does, and why you advice to do it? –  Fabien Jun 14 '13 at 12:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.