I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.

Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.

UPDATE: I have added the --update flag and usage seems mostly unchanged

#! /bin/bash


inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | (
while true; do
    read -t 1 LINE;
    if test -z "$LINE"; then
        if test ! -z "$WAITING"; then
                echo "CHANGE";
                rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/
  • I think I would try splitting this into 2 pieces to determine where the poor performance is actually coming from. If you run just the inotify piece by itself, w/o the rsync does the poor performance continue, or are you positive the root cause is rsync? – slm Dec 5 '13 at 14:08
  • Pretty sure it's the rsync. When I watch top that's what causes the increase in CPU, running as a system process. – kyleredon Dec 6 '13 at 15:27
  • Hm both scripts ignore the actual events. So this triggers a full rsync run. It would maybe be good to actually trigger rsync on partial trees. Or maybe add a time-limit - or collect multiple events in short time together and only kick off a new rsync at the end of a burst. – eckes Oct 12 '15 at 17:13

If the server has a slow processor avoid checksums and compression with rsync. I would remove ht "-z" option in the rsync command.

rsync --update -alvr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/

Note that it will not avoid rsync to compare the 650k files. You could rsync subdirectories of /var/www one by one to reduce the number of files checked at one time.

  • I ended up removing the compression option (z) and upping the WAITING var to 10 (seconds). This seems to have helped, rsync still spikes CPU load but it is shorter lived. Thanks – kyleredon Dec 17 '13 at 12:48

Thank you for posting your answer - it really helped me. I think you can simplify the loop:


sync() {
  rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/

watch() {
  inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' 

watch | (
while true ; do
  read -t 1 LINE && sync

You may need to adjust that slightly but the gist should be clear. I'm using a variant with Amazon's AWS s3 sync, so I may have substituted your rsync command in incorrectly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.