Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have several directories/web site resources, each has total size of several gigabytes. They are more than 90% the same. Sometimes only a few files are modified.
I need to download these files for backup purposes periodically. Storage space is no concern, but it is taking too long to download.

I would like to:
Use one such directory as reference (d1).
1. For another directory(d2), list all files that are identical between d1 and d2, save it into common.txt. Identical means relative path(within d1 and d2), filename and contents are the same, ignoring the owner/group and timestamps.
2. Copy all files, that are in d2 but not in d1 (includes different files) into d3.

Then I will download tarballed d1, d3 and common.txt to another machine.

Copy d3 to d2.
3. Copy files listed in common.txt from d1 to d2.

Could you please suggest how to resolve steps (1), (2), (3)?

share|improve this question
up vote 2 down vote accepted

Consider deduplication. Is there anything that modifies those files directly on your server (instead of deleting and uploading something fresh)? If there is not, you could symlink or hardlink the identical files outright. That would save storage space on your server, and also automatically solve your download problem, when using rsync -H which supports hardlinks. As long as you are aware of hardlink pitfalls (inplace modification for one modifies all of them, as they literally are the same file) that's the nicest solution by far.

You can use fdupes to find duplicate files efficiently. That covers duplicates content-wise, unfortunately there is no requirement for the filename. You could either use that and then filter the list of duplicates for filenames; or you could do the comparison yourself. Using find, stat, cmp.

Completely unrefined example:

cd d1
find -type f -exec ./compare.sh d1/{} d2/{}


if [ -f "$1" -a -f "$2" ]
    asize=`stat -c%s "$1"`
    bsize=`stat -c%s "$2"`

    if [ "$asize" == "$bsize" ]
        if cmp "$1" "$2"
            echo IDENTICAL "$1" "$2"
            echo DIFFERENT "$1" "$2"

Since you mentioned tarballs, instead of simply echoing whether a file is identical or different, you could just echo them into an include/exclude file list for use with tar --files-from / --exclude-from.

share|improve this answer
rsync seems to do the job. link – Mirbek Samiev Mar 12 '13 at 4:10

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.