38

I usually do rsync -n before doing an actual rsync. The thing is, how could I automate the second run to reuse exactly the list generated with --dry-run?

I mean, if I run rsync -n then I get a simulation of what rsync would do, my idea would be to take that output, process it then somehow make rsync only --include-from that output, so the full list has to be calculated only once.

If this doesn't even make sense, please, let me know.

By the way, I tend to use rsync -hva --progress --stats --delete --exclude-from "$EXCLUDEFILE". If you also have a suggestion on what options would you use, please, let me know. I'm trying to duplicate the contents of an external hard drive (that only contains images, videos and documents) to another external hard drive.

3
  • You could try redirecting the output from the --dry-run command into a file (e.g., rsync --dry-run -avz SOURCE DEST > include.txt), removing all lines that aren't files/paths from include.txt, and running rsync --include-from include.txt.
    – ThatGuy
    Feb 9, 2016 at 1:06
  • That last command should be rsync --include-from include.txt --exclude=* SOURCE DEST. See stackoverflow.com/a/19296415/1765910 for details.
    – ThatGuy
    Feb 9, 2016 at 1:18
  • Yes, that's the idea, but I don't know how to actually do that (plus I would like the approval from someone that actually knows what (s)he's doing). I just use rsync but don't know anything from shell scripts / bash…
    – Manuel
    Feb 9, 2016 at 16:34

1 Answer 1

33

It is certainly possible to use the file list generated during a dry run as an include file. Ideally, however, the extra lines at the top and bottom of the dry-run output should be deleted.

Example output:

sending incremental file list
[LIST OF FILES]

sent 226 bytes  received 34 bytes  520.00 bytes/sec
total size is 648,373,274  speedup is 2,493,743.36 (DRY RUN)

To remove these lines, pipe the rsync output into head and into tail. Then, save the processed output to an include file:

rsync --dry-run -avz source/ destination/ | head --lines=-3 | tail --lines=+3 > include.txt

Finally, tell rsync to use the generated include file (adding additional options as desired):

rsync --include-from=include.txt --exclude=* source/ dest/

4
  • 1
    The problem with this is that it doesn't cope with files that should be deleted. And is that method of tail/head robust enough to ensure having only the list of files?
    – Manuel
    Feb 10, 2016 at 22:18
  • Adding --delete causes files deleted from the source directories to be deleted from the destination directories as well, at least on the computer I tried it on. You can leave out the tail/head part entirely. rsync interprets each line in the include file as a file name and simply ignores non-existing files, so nothing will happen if you keep the extra lines (unless you happen to have files named "sending incremental file list", etc., in which case these files will be copied as well).
    – ThatGuy
    Feb 11, 2016 at 1:32
  • 1
    You also may use | sed '1d;/^$/,$d as an alternative for | head --lines=-3 | tail --lines=+3 . In case you want to add more filtering with sed it would be one command, not three.
    – Alek
    Jan 19, 2020 at 22:33
  • I am not sure this would make any sense as rsync works on a server / client base and the server will have to calculate the files / chunks to be transmitted anyway, be it all in one go or one after another. Would be nice to have an opinion about this from somebody with more knowledge than my humble self ;)
    – runlevel0
    Jan 8, 2021 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.