5

When running rsync on a directory with lots of files and directories at multiple levels, can we estimate the amount of work or time to finish?

  • the progress option only shows the progress of transferring a single file, not the progress of transferring all files and directories under the source directory.

  • One way I guess is to look at the what directories it has transferred so far, and compare that to the source.

    It will help a lot if I know the order of files and directories in which rsync transfers them.

    I guess that it may be related to that rsync runs multiple threads and what each thread does?

    I am not sure what order it chooses, and my previous guess of bread-first order seems not correct (so I strike it through).

3

You can make rsync print one line per file using -i and then use pv -l to report progress based on line count (in effect file count).

You will need pv (pipe viewer): http://www.ivarch.com/programs/pv.shtml

rsync -ai sourcedir/ targetdir/ | pv -l -s filecount > logfile

Use the following command to get file count:

find sourcedir | wc -l

Note: this command will show progress information based on number of files copied. This works best if there are many smallish files. If there are only a few files which are huge then you will not have much fun.


To see progress when you are updating (or comparing) an existing copy:

(more information: Compare directories but not content of files)

rsync -aii --delete sourcedir/ targetdir/ | pv -l -s filecount > logfile

The second -i makes rsync print one line per file even if they are equal.

Add -n to compare (not actually copy or delete anything).

Leave out --delete as needed.

This command will print to screen in real time the files that differ:

rsync -aii --delete sourcedir/ targetdir/ | pv -l -s filecount | 
    tee logfile | grep -v "^\."

The commands above work best when there are many smallish files. If you have few huge files then some of the rsync built in progress reports might be of more help. See the rsync man page for -P or --progress or --info=progress2. But beware: these options will not work well with pv. Or at least I have not found out how.

Here is a crude workaround to see progress based on size:

  • note the free space of target partition before copying using df -h.
  • note the size of the source dir using du -sh.
  • use watch df -h and watch the size grow.

Obviosly this only works when copying and not when updating or comparing.

1

You can use $ time rsync * /temp When you want to know estimate time.You should check total size of files before you did rsync. Fast or slow of rsync depended on your network when you did rsync thru network. You can use $ ls -ltr to check which directories have been backup it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.