Is it possible to compare two directories with rsync and only print the differences? There's a dry-run option, but when I increase verbosity to a certain level, every file compared is shown.

ls -alR and diff is no option here, since there are hardlinks in the source making every line different. (Of course, I could delete this column with perl.)

up vote 41 down vote accepted

You will propably have to run something like rsync -avun --delete in both directions.

But what are you actually trying to accomplish?

Update:

rsync -avun --delete $TARGET $SOURCE |grep "^deleting " will give you a list of files that do not exist in the target-directory.

"grep delet" because each line prints : deleting ..file..

rsync -avun $SOURCE $TARGET will give you a list of "different" files (including new files).

  • I want to list all files that are in the source but not in the target directory. All modified files should be listed too. – chris Dec 3 '12 at 18:55
  • @chris I updated my answer. Is this what you are searching for? – Nils Dec 4 '12 at 21:26
  • Yes that's it. Thank you. – chris Dec 6 '12 at 17:23
  • 1
    The grep should be on "^delet" as that text string appeared in a few dozens of files from a RAID I am copying. But only 11 started with deleting – Chris K Sep 27 '15 at 23:13
  • 1
    @Chris K right you are. – Nils Sep 29 '15 at 19:59

To add to Nils's answer (for anyone coming across this via Google), by default rsync only compares the file sizes and modification times to tell if there are any differences. (If those are different it does more, but if they're the same, it stops there.)

If you want to compare actual file contents, even for files which have the same size and last modification time, add the flag -c to tell rsync to compare the files using a checksum.

rsync -avnc $SOURCE $TARGET

(The -u option tells rsync to ignore files which are newer in $TARGET than on $SOURCE, which you probably don't want if you're comparing contents.)

  • 2
    If you only care that the data is the same you might want to add --no-group --no-owner --no-perms --no-times or some combination of these based on your needs. – flungo Sep 30 '16 at 17:31
  • 1
    @flungo, or just use a subset of the options implied by -a instead of -a, e.g. rsync -rlDcnv --delete $SOURCE $TARGET – maxschlepzig Nov 26 '16 at 22:29
  • Please add --delete to list files only existing in $TARGET – Tom Hale Oct 31 '17 at 5:42

Just for those less familiar with rsync:

rsync -rvnc --delete ${SOURCE}/ ${DEST}
  • -n : most important bit -- do not change anything ;
  • -rc : compare only the contents (otherwise use -ac) ;
  • -v : list the files )
  • --delete : look for a symmetrical, not a uni-directional difference.
  • Finally, / means "look inside the directory, and compare its contents to the destination".

It will print a usual rsync output,

  • with one <filename> on a line for every "new" file in ${SOURCE}
  • and one "deleting <filename>" line for each "new" file in ${DEST}.

  • It may also print a few warnings, like "skipping non-regular file <filename>" for symlinks.

PS. I know it's a terrible PS -- but it was indeed added in a rush. Nevertheless, I bet one may find this useful.


PPS. Alternatively, one could also do

find $SOURCE -type f -exec md5sum {} \; | tee source.md5
find $DEST   -type f -exec md5sum {} \; | tee dest.md5

If the filenames do not contain newlines, we can then sort both *.md5 files, and diff them. ( This will work only for files, though; that is, an empty directory on either side won't be detected. )

I understand from your question that you do not want to use diff on ls, but you can also use diff recursively on directories:

diff -rq DIR1 DIR2

It took me a few tries to get this to work. Nils' answer requires that $TARGET ends in a trailing /, as explained by ジョージ.

Here is a version that explicitly adds the trailing /:

rsync -avun --delete ${TARGET}/ ${SOURCE}  | sed -ne 's/^deleting *//p'

This gives the list of files that exist below the ${SOURCE} directory but not below the ${TARGET} directory.

I use sed here to remove the leading deleting from the output lines, and to print only those lines.

I do not use the rsync option -c because comparing file contents would be much slower for my use cases, and comparing only file sizes and modification times also seems sufficient in these cases. I have no reason to suspect that my computers suffer from clock skew problems or that something maliciously changed time stamps. Also, the outcome of -c cannot change the decision to delete a file, only the decision to update or keep a file.

I also use -u and -a (rather than -r), so that I can later re-use the command line and change it to copy selected directories and files from ${SOURCE} to ${TARGET}, like this:

rsync -avu ${SOURCE}/{dirA,dirB,fileX} ${TARGET}   # copy some files

Surprisingly no answer in 6 years uses the -i option or gives nice output so here I'll go:

TLDR - Just show me the commands

rsync -rin --ignore-existing $LEFT_DIR/ $RIGHT_DIR/|sed -e 's/^[^ ]* /L             /'
rsync -rin --ignore-existing $RIGHT_DIR/ $LEFT_DIR/|sed -e 's/^[^ ]* /R             /'
rsync -rin --existing $LEFT_DIR/ $RIGHT_DIR/|sed -e 's/^/X /'

Understanding the output

Here's an example of the output:

L             file-only-in-Left-dir
R             file-only-in-right-dir
X >f.st...... file-with-dif-size-and-time
X .f...p..... file-with-dif-perms

Note the first character of every line:

  • L/R mean that the file/dir appears only at the Left or Right dir.
  • X means that the file appears on both sides but is not the same (in which case the next 11 characters give you more info. s,t and p depict differences in size, time and permissions respectively -- for more info try man rsync and search for --itemize-changes).

Extra options you may wish to use

If you want to also compare the owner/group/permissions of the files add the options -o/-g/-p respectively. Finally note that by default rsync considers two files the same if they have the same name, time and size. This is extremely fast and most of the times more than enough but if you want to be 100% sure add -c to also compare the contents of files with the same name, time & size.

TLDR - Just give me a script to call

Here it is. Call it like this

diff-dirs Left_Dir Right_Dir [options]

All options mentioned above in section "Extra options you may wish to use" also apply here.

#!/bin/bash
# Compare two directories using rsync and print the differences
# CAUTION: options MUST appear after the directories
#
# SYNTAX
#---------
# diff-dirs Left_Dir Right_Dir [options]
#
# EXAMPLE OF OUTPUT
#------------------
# L             file-only-in-Left-dir
# R             file-only-in-right-dir
# X >f.st...... file-with-dif-size-and-time
# X .f...p..... file-with-dif-perms
#
# L / R mean that the file/dir appears only at the `L`eft or `R`ight dir. 
#
# X     means that a file appears on both sides but is not the same (in which
#       case the next 11 characters give you more info. In most cases knowing
#       that s,t and p depict differences in Size, Time and Permissions 
#       is enough but `man rsync` has more info
#       (look at the --itemize-changes option)
#
# OPTIONS
#---------
# All options are passed to rsync. Here are the most useful for the purpose
# of directory comparisons:
#
# -c will force comparison of file contents (otherwise only
#    time & size is compared which is much faster)
#
# -p/-o/-g will force comparison of permissions/owner/group

LEFT_DIR=$1; shift
RIGHT_DIR=$1; shift
OPTIONS="$*"

# Files that don't exist in Right_Dir
rsync $OPTIONS -rin --ignore-existing $LEFT_DIR/ $RIGHT_DIR/|sed -e 's/^[^ ]* /L             /'
# Files that don't exist in Left_Dir
rsync $OPTIONS -rin --ignore-existing $RIGHT_DIR/ $LEFT_DIR/|sed -e 's/^[^ ]* /R             /'
# Files that exist in both dirs but have differences
rsync $OPTIONS -rin --existing $LEFT_DIR/ $RIGHT_DIR/|sed -e 's/^/X /'

How does it work?

The core of the job is performed by calling rsync like this:

rsync -rin --ignore-existing $DIR_A/ $DIR_B/

We use: -i to tell rsync to print one line of output for every file formated in a special way*, -n to suppress normal behavior of rsync (to try to copy/delete/sync the two dirs), -rto work recursively for all files/sub-dirs.

We call rsync three times:

1st call: print files that don't exist in Dir_B. We need to use --ignore-existing to ignore files that exist on both sides.

rsync -rin --ignore-existing $DIR_A/ $DIR_B/

2nd call: Exactly as before but we swap the order of DIR_A/DIR_B.

3rd call: Finally we use --existing to only check files that appear in both dirs.

rsync -rin --existing $DIR_A/ $DIR_B/

Notes

*: Use man rsync and look for --itemize-changes

I have another idea of doing this:

rsync -rn --out-format=FILEDETAIL::%n  $TARGET $SOURCE  | grep "^FILEDETAIL"

You can match "FILEDETAIL::" with the output of the command. Also, you can change the string "FILEDETAIL::". The "%n" is the file name.

-r This tells rsync to copy directories recursively.

-n This make rsync perform a trial run that doesn't make any changes.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.