Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

In Ubuntu, I want to copy a big file from my hard drive to a removable drive by rsync. For some other reason, the operation cannot complete in a single run. So I am trying to figure out how to use rsync to resume copying the file from where it left off last time.

I have tried to use the option --partial or --inplace, but together with --progress, I found rsync with --partial or --inplace actually starts from the beginning instead of from what was left last time. Mannually stopping rsync early and checking the size of the received file also confirmed what I found.

But with --append, rsync starts from what was left last time.

I am confused as I saw on the manpage --partial, --inplace, and --append seem to relate to resuming copying from what was left last time. Is someone able to explain the difference? Why don't --partial or --inplace work for resuming copying? Is it true that for resuming copying, rsync has to work with the --append option?

Also, if a partial file was left by mv or cp, not by rsync, will rsync --append correctly resume copying the file?

Thanks and regards!

share|improve this question
add comment

4 Answers 4

up vote 13 down vote accepted

To resume an interrupted copy, you should use rsync --append. From the man page's explanation of --append:

This causes rsync to update a file by appending data onto the end of the file, which presumes that the data that already exists on the receiving side is identical with the start of the file on the sending side. [...] Implies --inplace, [...]

Option --inplace makes rsync (over)write the destination file contents directly; without --inplace, rsync would:

  1. create a new file with a temporary name,
  2. copy updated content into it,
  3. swap it with the destination file, and finally
  4. delete the old copy of the destination file.

The normal mode of operation mainly prevents conflicts with applications that might have the destination file open, and a few other mishaps which are duly listed in the rsync manpage.

Note that, if a copy/update operation fails in steps 1.-3. above, rsync will delete the temporary destination file; the --partial option disables this behavior and rsync will leave partially-transferred temporary files on the destination filesystem. Thus, resuming an a single file copy operation will not gain much unless you called the first rsync with --partial or --partial-dir (same effect as --partial, in addition instructs rsync to create all temporary files in a specific directory).

share|improve this answer
    
Thanks! If a partial file was left by mv or cp not by rsync, will rsync --append correctly resume the file copying? –  Tim Sep 26 '10 at 19:05
    
@Tim In short, --append makes rsync believe that, if two corresponding files have different length, then the shorter one is identical to the initial part of the longer one. So, yes, if you start copying a large file with cp and interrupt the copy process, then rsync --append will copy only the remaining part of the file. (Note: if cp is interrupted by a system crash, there is a small chance that the file contents and metadata are not in sync, i.e., the file is corrupted. In this case, running rsync once more without --append should fix the problem.) –  Riccardo Murri Sep 26 '10 at 20:05
    
So If I understand this correctly, there is no way to tell rsync to verify a partial file and resume transfer to that partially transferred file? –  Winny 2 days ago
add comment

David Schwartz is correct, --partial (or better, -P) does do what you want. I verified this on a 37G file that was stopped ~8g into it, over a network. rsync quickly scanned the first parts of the partial (showing progress as it was going thanks to -P), and then resumed the transfer to the end of the partial file.

share|improve this answer
add comment

You were doing it right --partial does what you want. It appears to be starting from the beginning because it always starts at the beginning of the list of file data chunks it needs to copy. The --append option is dangerous and will result in a corrupt file if the data does not match for some reason.

share|improve this answer
add comment

Be aware that --append implies --inplace, which itself implies --partial.

  • By just using --partial you should cause rsync to leave partial transfers and resume them in subsequent attempts.

  • By using --append you should cause rsync to both leave partial files and resume them next time. After transfer rsync should verify the checksum of transmitted data only.

  • --append-verify includes the whole file in the checksum verification, including any portion transferred in a previous transfer.

  • With either --append or --append-verify a failed checksum verification should cause the file to be re-transmitted completely (using --inplace)

You should be able to resume a mv or cp operation with rsync but you may want to use the --append-verify option for peace of mind.

I have also been told that using --append causes rsync to copy only those files which have been partially copied already. This is not my experience but may depend on rsync version.

More info here

share|improve this answer
    
--inplace implies --partial not the other way around. –  TomOnTime Jul 5 at 15:12
    
@TomOnTime thank you, I have edited my answer. –  TomG Jul 7 at 14:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.