I'm using rsync in a script (on OSX) to copy from a raid to 2 backup drives, which happens when I run the script, probably 10x-20x in a day. I'm trying to figure out a way to parse the rsync log file, or anything for that matter, to tell me the number of files that were copied. Here is my rsync command:

rsync -auvh --delete --progress "$SourceTx" "$Dest1Tx" --log file="$SourceRoot/""CopyLog1.txt"

and my attempt (not working) to grep the log-file to derive the # of files copied:

cd "$SourceRoot/"
howMany=$(cat CopyLog1.txt | grep -c .mov)
osascript -e 'display notification "Transcode Copy Complete, '$howMany' files transfered, Log Files Created" with title "Transcode Copy Script"'

To get the number of files transferred, use the --stats option. For example:

$ rsync -auvh --stats --delete --log-file=1.log --progress "$SourceTx" "$Dest1Tx" | grep 'files transferred'
Number of regular files transferred: 21

To capture the number of files transferred to a shell variable:

nfiles=$(rsync -auvh --stats --delete --log-file=1.log --progress "$SourceTx" "$Dest1Tx" | awk '/files transferred/{print $NF}')

The above was filtered to just show the number of files transferred. To illustrate the other information available with --stats:

$ rsync -auh --stats --delete  srcdir destdir

Number of files: 14 (reg: 10, dir: 4)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 209 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 342
Total bytes received: 27

sent 342 bytes  received 27 bytes  738.00 bytes/sec
total size is 209  speedup is 0.57
  • this line: nfiles=$(rsync -auvh --stats --delete --log-file=1.log --progress "$SourceTx" "$Dest1Tx" | awk '/files transferred/{print $NF}') runs rsync and then stores the number of files transferred into the variable nfiles correct? So in my notification I would just call $nfiles and be done? Also, if I call --stats, will this print in the log file? Thanks btw. much appreciated. – borneo1910 Sep 28 '16 at 6:26
  • @borneo1910 Yes, nfiles would have the number of files transferred and you can use that later in the script has you like. (Technically, you don't "call it" because it is not a function or executable; it is just a variable with a value.) Also, yes, the stats information appears in the log file. – John1024 Sep 28 '16 at 6:36
  • I lost the progress output, ie % copied per file and data rate, etc. is there a way to get that back? – borneo1910 Sep 28 '16 at 20:41
  • If you still want to see all the progress information, add a tee command. For example: nfiles=$(rsync -auvh --stats --delete --log-file=1.log --progress "$SourceTx" "$Dest1Tx" | tee /dev/stderr | awk '/files transferred/{print $NF}') – John1024 Sep 28 '16 at 22:15
  • Ok that works very well, but now here's a question. Something slowed down the transfer considerably. Drag and drop is about 10x faster. Any thoughts? – borneo1910 Sep 29 '16 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.