I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.

Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.

UPDATE: I have added the --update flag and usage seems mostly unchanged

#! /bin/bash


inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | (
while true; do
    read -t 1 LINE;
    if test -z "$LINE"; then
        if test ! -z "$WAITING"; then
                echo "CHANGE";
                rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/
  • 1
    I think I would try splitting this into 2 pieces to determine where the poor performance is actually coming from. If you run just the inotify piece by itself, w/o the rsync does the poor performance continue, or are you positive the root cause is rsync?
    – slm
    Dec 5, 2013 at 14:08
  • Pretty sure it's the rsync. When I watch top that's what causes the increase in CPU, running as a system process.
    – kyleredon
    Dec 6, 2013 at 15:27
  • Hm both scripts ignore the actual events. So this triggers a full rsync run. It would maybe be good to actually trigger rsync on partial trees. Or maybe add a time-limit - or collect multiple events in short time together and only kick off a new rsync at the end of a burst.
    – eckes
    Oct 12, 2015 at 17:13
  • This script seems to run full rsync every time a single file has been changed. If/when your inotifywait returns multiple lines, the whole rsync will be run equally many times. You either need to collect list of modified files and sync that list as soon as possible (basically start sync on first modified file and start collecting list of new files in paraller with the sync, re-sync with the new list after the first sync is ready and start collecting another list for the time sync is going). Or just run unison instead. Sep 18, 2020 at 10:41
  • @kyleredon, you might remove MODIFY event, since it will be generated A LOT when a large file is being downloaded/created. Also, if you act/sync on MODIFY event, without waiting for CLOSE_WRITE event, you might sync an incomplete file. In essence, you only need to trigger sync on these events: CLOSE_WRITE, MOVED_TO, MOVED_FROM, DELETE.
    – Mladen B.
    Feb 11, 2021 at 11:01

3 Answers 3


If the server has a slow processor avoid checksums and compression with rsync. I would remove ht "-z" option in the rsync command.

rsync --update -alvr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/

Note that it will not avoid rsync to compare the 650k files. You could rsync subdirectories of /var/www one by one to reduce the number of files checked at one time.

  • I ended up removing the compression option (z) and upping the WAITING var to 10 (seconds). This seems to have helped, rsync still spikes CPU load but it is shorter lived. Thanks
    – kyleredon
    Dec 17, 2013 at 12:48

Thank you for posting your answer - it really helped me. I think you can simplify the loop:


sync() {
  rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/

watch() {
  inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' 

watch | (
while true ; do
  read -t 1 LINE && sync

You may need to adjust that slightly but the gist should be clear. I'm using a variant with Amazon's AWS s3 sync, so I may have substituted your rsync command in incorrectly.


You might check this tool syncthing

I'm using dockerized version from this repo

I have 411k files to sync using this tool, I had some issues regarding file names encoding for some files named in arabic

docker run -d --rm \
  --name=syncthing \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Asia/Jordan \
  -e UMASK_SET=022 \
  -p 8384:8384 \
  -p 22000:22000 \
  -p 21027:21027/udp \
  -v "$(pwd)/config:/config" \
  -v "path_to_sync:/config/Sync" \
  --net host \

Note: use GUI to configure

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .