1

I have been backing up a server over rsync (v3.0.6) for a while. I noticed that it takes about 20 minutes just to get the incremental file list (I am using the whole file flag). I am excluding three files, if I take these out it only takes a second to start syncing. There are only 319 files (689 if you count the directories that get skipped). Any thoughts or ideas? Pattern matching shouldn't be this slow?

rsync -vvnWr --exclude='aaa.tar.gz' --exclude='bbb.tar.gz' -exclude='ccc.tar.gz' --include='*.tar.gz' --exclude='*' --delete -e "ssh -p 34577 -i /root/.ssh/backup_rsa" root@xxx.xxx.xxx.xxx:/backup/cpbackup/daily/ /backup/jarvis/cpanel/$((($(date +\%-d)-1)/7+1))/*

Sample output

[sender] showing file ddd.tar.gz because of pattern *.tar.gz
[sender] showing file eee.tar.gz because of pattern *.tar.gz
[sender] hiding directory fff.0 because of pattern *
[sender] showing file ggg.tar.gz because of pattern *.tar.gz
...

The output is the same with our without exclusions, obviously it skips the ones I list. With exclusions there are long pauses between each of these. I changed the IP and filenames.

Thanks!

  • Why wholefile rather than the default incremental? – roaima Apr 23 '18 at 22:15
1

I just figured it out, must have needed to sleep on it. I am missing one of the hyphens in an exclude. It must have been reading it as additional parameters without causing any syntax errors, with the -c checksum flag being the one probably causing it to slow down.

Thanks everyone for reading!

  • That's an interesting one. The first letter of -exclude takes the remaining text as its argument (up to the space). If you hadn't specified -e further along the command you'd have seen the error immediately (rsync: Failed to exec xclude=ccc.tar.gz: No such file or directory (2)). Oh well. – roaima Apr 24 '18 at 13:43
0

The main reason this is so slow is because you've disabled most of the optimisations that rsync has to offer. You're almost at the stage of copying all the included files on every run.

Why? You've mandated --whole-file instead of allowing rsync to generate differences. You're also not copying the timestamp with -t or -a, so rsync has to read each file on the local and remote systems to determine if the checksums match.

Try this instead

rsync --dry-run -vvtr \
    --exclude='aaa.tar.gz' --exclude='bbb.tar.gz' -exclude='ccc.tar.gz' --include='*.tar.gz' --exclude='*' \
    --delete --delete-during \
    -e "ssh -p 34577 -i /root/.ssh/backup_rsa" \
        root@xxx.xxx.xxx.xxx:/backup/cpbackup/daily/ \
        /backup/jarvis/cpanel/$((($(date +\%-d)-1)/7+1))/
  • Thanks for the response! The files are different every time because they are the daily backups for a web/database server. It doesn't explain the reasons its slow only when the excludes are there. I am 100% fine with the speed after that. Do you have any other thoughts with this additional info? Thanks! – Kyle Eadie Apr 24 '18 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.