121

Is it possible to compare two directories with rsync and only print the differences? There's a dry-run option, but when I increase verbosity to a certain level, every file compared is shown.

ls -alR and diff is no option here, since there are hardlinks in the source making every line different. (Of course, I could delete this column with perl.)

1

7 Answers 7

78

To add to Nils's answer (for anyone coming across this via Google), by default rsync only compares the file sizes and modification times to tell if there are any differences. (If those are different it does more, but if they're the same, it stops there.)

If you want to compare actual file contents, even for files which have the same size and last modification time, add the flag -c to tell rsync to compare the files using a checksum.

rsync -avnc $SOURCE $TARGET

(The -u option tells rsync to ignore files which are newer in $TARGET than on $SOURCE, which you probably don't want if you're comparing contents.)

5
  • 11
    If you only care that the data is the same you might want to add --no-group --no-owner --no-perms --no-times or some combination of these based on your needs.
    – flungo
    Sep 30, 2016 at 17:31
  • 4
    @flungo, or just use a subset of the options implied by -a instead of -a, e.g. rsync -rlDcnv --delete $SOURCE $TARGET Nov 26, 2016 at 22:29
  • 2
    Please add --delete to list files only existing in $TARGET
    – Tom Hale
    Oct 31, 2017 at 5:42
  • 2
    You need to add a / at the end of source but NOT target: rsync -avnc $SOURCE/ $TARGET
    – Wildcard
    Sep 8, 2023 at 22:54
  • If you only want the names of the changed files & directories, but not the auxiliary output of rsync ("sending incremental file list", etc), then use instead of -v the more granular option --info=NAME
    – halloleo
    Oct 31, 2023 at 23:53
66

You will propably have to run something like rsync -avun --delete in both directions.

But what are you actually trying to accomplish?

Update:

rsync -avun --delete $TARGET $SOURCE |grep "^deleting " will give you a list of files that do not exist in the target-directory.

"grep delet" because each line prints : deleting ..file..

rsync -avun $SOURCE $TARGET will give you a list of "different" files (including new files).

1
  • 1
    note regarding rsync, when it compare $SOURCE to $TARGET it doesn't mean the two directories are identical, it just compare what is found $SOURCE to the $TARGET, so if your rsync stopped in the middle, make sure to compare SOURCE to TARGET vs TARGET to SOURCE ...
    – Ricky Levi
    Jul 3, 2021 at 7:18
63

Just for those less familiar with rsync:

rsync -rvnc --delete ${SOURCE}/ ${DEST}
  • -r : recurse into directories ;
  • -v : list the files )
  • -n : most important bit -- do not change anything ;
  • -c : compare the contents via checksum not not mod-time & size (use -a otherwise) ;
  • --delete : look for a symmetrical, not a uni-directional difference.
  • Finally, / means "look inside the directory, and compare its contents to the destination".

It will print a usual rsync output,

  • with one <filename> on a line for every "new" file in ${SOURCE}
  • and one "deleting <filename>" line for each "new" file in ${DEST}.
  • It may also print a few warnings, like "skipping non-regular file <filename>" for symlinks.

PS. I know it's a terrible PS -- but it was indeed added in a rush. Nevertheless, I bet one may find this useful.


PPS. Alternatively, one could also do
find $SOURCE -type f -exec md5sum {} \; | tee source.md5
find $DEST   -type f -exec md5sum {} \; | tee dest.md5

If the filenames do not contain newlines, we can then sort both *.md5 files, and diff them. ( This will work only for files, though; that is, an empty directory on either side won't be detected. )

1
  • 7
    The / in SOURCE is important ... and its absence in DEST also ! Jan 16, 2021 at 12:30
59

Surprisingly no answer in 6 years uses the -i option or gives nice output so here I'll go:

TLDR - Just show me the commands

rsync -rin --ignore-existing "$LEFT_DIR"/ "$RIGHT_DIR"/|sed -e 's/^[^ ]* /L             /'
rsync -rin --ignore-existing "$RIGHT_DIR"/ "$LEFT_DIR"/|sed -e 's/^[^ ]* /R             /'
rsync -rin --existing "$LEFT_DIR"/ "$RIGHT_DIR"/|sed -e 's/^/X /'

Understanding the output

Here's an example of the output:

L             file-only-in-Left-dir
R             file-only-in-right-dir
X >f.st...... file-with-dif-size-and-time
X .f...p..... file-with-dif-perms

Note the first character of every line:

  • L/R mean that the file/dir appears only at the Left or Right dir.
  • X means that the file appears on both sides but is not the same (in which case the next 11 characters give you more info. s,t and p depict differences in size, time and permissions respectively -- for more info try man rsync and search for --itemize-changes).

Extra options you may wish to use

If you want to also compare the owner/group/permissions of the files add the options -o/-g/-p respectively. Finally note that by default rsync considers two files the same if they have the same name, time and size. This is extremely fast and most of the times more than enough but if you want to be 100% sure add -c to also compare the contents of files with the same name, time & size.

TLDR - Just give me a script to call

Here it is. Call it like this

diff-dirs Left_Dir Right_Dir [options]

All options mentioned above in section "Extra options you may wish to use" also apply here.

#!/bin/bash
# Compare two directories using rsync and print the differences
# CAUTION: options MUST appear after the directories
#
# SYNTAX
#---------
# diff-dirs Left_Dir Right_Dir [options]
#
# EXAMPLE OF OUTPUT
#------------------
# L             file-only-in-Left-dir
# R             file-only-in-right-dir
# X >f.st...... file-with-dif-size-and-time
# X .f...p..... file-with-dif-perms
#
# L / R mean that the file/dir appears only at the `L`eft or `R`ight dir. 
#
# X     means that a file appears on both sides but is not the same (in which
#       case the next 11 characters give you more info. In most cases knowing
#       that s,t,T and p depict differences in Size, Time and Permissions 
#       is enough but `man rsync` has more info
#       (look at the --itemize-changes option)
#
# OPTIONS
#---------
# All options are passed to rsync. Here are the most useful for the purpose
# of directory comparisons:
#
# -c will force comparison of file contents (otherwise only
#    time & size is compared which is much faster)
#
# -p/-o/-g will force comparison of permissions/owner/group

if [[ -z $2 ]] ; then
    echo "USAGE: $0 dir1 dir2 [optional rsync arguments]"
    exit 1
fi

set -e

LEFT_DIR=$1; shift
RIGHT_DIR=$1; shift
OPTIONS="$*"

# Files that don't exist in Right_Dir
rsync $OPTIONS -rin --ignore-existing "$LEFT_DIR"/ "$RIGHT_DIR"/|sed -e 's/^[^ ]* /L             /'
# Files that don't exist in Left_Dir
rsync $OPTIONS -rin --ignore-existing "$RIGHT_DIR"/ "$LEFT_DIR"/|sed -e 's/^[^ ]* /R             /'
# Files that exist in both dirs but have differences
rsync $OPTIONS -rin --existing "$LEFT_DIR"/ "$RIGHT_DIR"/|sed -e 's/^/X /'

How does it work?

We're calling rsync like this:

rsync -rin ...

We use -i (--itemize-changes) to tell rsync to print one line of output for every file that contains information about any differences between the two directories. We need -n to suppress normal behavior of rsync (which is to try to sync the two dirs by copying/deleting files). we also need -rto work recursively for all files/sub-dirs.

We call rsync three times:

1st call: print files that don't exist in Dir_B. We need to use --ignore-existing to ignore files that exist on both sides.

rsync -rin --ignore-existing $DIR_A/ $DIR_B/

2nd call: Exactly as before but we swap the order of DIR_A/DIR_B.

3rd call: Finally we use --existing to only check files that appear in both dirs.

rsync -rin --existing $DIR_A/ $DIR_B/
8
  • 4
    Don't know about others, but I am using your script. Great work! thanks
    – Marinaio
    Apr 19, 2019 at 13:45
  • Thank you so much! I needed some tweaks, will share them below in case someone is looking for something similar. First, I wanted to run remote rsync as sudo user, for that I added --rsync-path="sudo rsync" to every rsync command. Second, I wanted to compare local directory with remote directory. I added --rsh "ssh -p1234" because in my case SSH is running on port 1234. Then I called script like diff-dirs [email protected]:/mnt/Vol1/dir1/ /localMnt/dir1 -c.
    – sen4ik
    Oct 31, 2019 at 4:22
  • 1
    Your intuition may be true Steven but it's been years since I came up with this solution (long before posting here) and I hardly remember the details. If you test on a sample with only the minimum files/dirs you'll find out easily.
    – ndemou
    Dec 2, 2019 at 6:24
  • 1
    @sen4ik I like the script and using it for an ssh directory. It is perfect except that it is asking for the psw 3 times for the 3 calls :<
    – Enissay
    Mar 2, 2020 at 17:13
  • 2
    Interesting script! One thing to be aware of: The script has problems with directories containing non-reguiar files: They seem to be listed in the left, right and changed section!
    – halloleo
    Apr 21, 2022 at 12:37
17

I understand from your question that you do not want to use diff on ls, but you can also use diff recursively on directories:

diff -rq DIR1 DIR2
2
  • This is a great answer. It works best in most cases, except 2: 1) if a directory is remote 2) if there are rsync filters to be applied (that is some subdirectories must be omitted from comparison). Mar 11, 2020 at 13:53
  • If you want to compare file contents, diff is good for that. If you want to trust sizes and mode-times, rsync can do that. Apr 17, 2023 at 3:36
3

It took me a few tries to get this to work. Nils' answer requires that $TARGET ends in a trailing /, as explained by ジョージ.

Here is a version that explicitly adds the trailing /:

rsync -avun --delete ${TARGET}/ ${SOURCE}  | sed -ne 's/^deleting *//p'

This gives the list of files that exist below the ${SOURCE} directory but not below the ${TARGET} directory.

I use sed here to remove the leading deleting from the output lines, and to print only those lines.

I do not use the rsync option -c because comparing file contents would be much slower for my use cases, and comparing only file sizes and modification times also seems sufficient in these cases. I have no reason to suspect that my computers suffer from clock skew problems or that something maliciously changed time stamps. Also, the outcome of -c cannot change the decision to delete a file, only the decision to update or keep a file.

I also use -u and -a (rather than -r), so that I can later re-use the command line and change it to copy selected directories and files from ${SOURCE} to ${TARGET}, like this:

rsync -avu ${SOURCE}/{dirA,dirB,fileX} ${TARGET}   # copy some files
0

I have another idea of doing this:

rsync -rn --out-format=FILEDETAIL::%n  $TARGET $SOURCE  | grep "^FILEDETAIL"

You can match "FILEDETAIL::" with the output of the command. Also, you can change the string "FILEDETAIL::". The "%n" is the file name.

As stated in rsync manpage:

-r, --recursive             recurse into directories

-n, --dry-run               perform a trial run with no changes made

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .