Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.

Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.

UPDATE: I have added the --update flag and usage seems mostly unchanged

#! /bin/bash

EVENTS="CREATE,DELETE,MODIFY,MOVED_FROM,MOVED_TO"

inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | (
WAITING="";
while true; do
    LINE="";
    read -t 1 LINE;
    if test -z "$LINE"; then
        if test ! -z "$WAITING"; then
                echo "CHANGE";
                WAITING="";
                rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/
        fi;
    else
        WAITING=1;
    fi;
done)
share|improve this question
    
I think I would try splitting this into 2 pieces to determine where the poor performance is actually coming from. If you run just the inotify piece by itself, w/o the rsync does the poor performance continue, or are you positive the root cause is rsync? –  slm Dec 5 '13 at 14:08
    
Pretty sure it's the rsync. When I watch top that's what causes the increase in CPU, running as a system process. –  kyleredon Dec 6 '13 at 15:27

2 Answers 2

up vote 3 down vote accepted

If the server has a slow processor avoid checksums and compression with rsync. I would remove ht "-z" option in the rsync command.

rsync --update -alvr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/

Note that it will not avoid rsync to compare the 650k files. You could rsync subdirectories of /var/www one by one to reduce the number of files checked at one time.

share|improve this answer
    
Thanks, will give that a try. –  kyleredon Dec 10 '13 at 8:01
    
I ended up removing the compression option (z) and upping the WAITING var to 10 (seconds). This seems to have helped, rsync still spikes CPU load but it is shorter lived. Thanks –  kyleredon Dec 17 '13 at 12:48

Thank you for posting your answer - it really helped me. I think you can simplify the loop:

EVENTS="CREATE,DELETE,MODIFY,MOVED_FROM,MOVED_TO"

sync() {
  rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/

}
watch() {
  inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' 
}


watch | (
while true ; do
  read -t 1 LINE && sync
done
)

You may need to adjust that slightly but the gist should be clear. I'm using a variant with Amazon's AWS s3 sync, so I may have substituted your rsync command in incorrectly.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.