18

With diff -r I can do this task, however it takes so long because diff checks file's content.

I want something that determine that two files are the same regarding of their size, last modified, etc. But no checking bit by bit the file (for example a video takes sooo long)

Is there any other way?

16

rsync, by default, compares only file metadata.

rsync -n -a -i --delete source/ target/

explanation:

  • -n do not actually copy or delete <-- THIS IS IMPORTANT!!1
  • -a compare all metadata of file like timestamp and attributes
  • -i print one line of information per file
  • --delete also report files which are not in source

note: it is important to append the directory names with a slash. this is an rsync thing.

if you also want to see lines printed for files that are identical then provide -i twice

rsync -n -a -ii --delete source/ target/

example output:

*deleting   removedfile   (file in target but not in source)
.d..t...... ./            (directory with different timestamp)
>f.st...... modifiedfile  (file with different size and timestamp)
>f+++++++++ newfile       (file in source but not in target)
.f          samefile      (file that has same metadata. only with -ii)

remember that rsync only compares metadata. that means if the file content changed but metadata stayed the same then rsync will report that file is same. this is an unlikely scenario. so either trust that when metadata is same then data is same, or you have to compare file data bit by bit.

bonus: for progress information see here: Estimate time or work left to finish for rsync?

  • The slashes in source/ and target/ are also both very important! (Without them, you will compare source and target directory names along with the child file names, so all file names will differ.) – peschü Feb 16 at 14:10
3

Use the -q (--brief) option with diff -r (diff -qr). From the info page for GNU diff:

1.6 Summarizing Which Files Differ

When you only want to find out whether files are different, and you don't care what the differences are, you can use the summary output format. In this format, instead of showing the differences between the files, diff' simply reports whether files differ. The--brief' (`-q') option selects this output format.

This format is especially useful when comparing the contents of two directories. It is also much faster than doing the normal line by line comparisons, because `diff' can stop analyzing the files as soon as it knows that there are any differences.

This will not compare line by line, but rather the file as a whole, which greatly speeds up the processor (what' you're looking for).

  • 1
    The problem of - q is that it compares normal and when finds a difference stops (if were normal mode it keeps comparing), so if huge files are the same it will last a lot. – yzT Dec 24 '12 at 18:34
2

Here's a quick python script that will check that the filenames, mtimes, and file sizes are all the same:

import os
import sys

def getStats(path):
    for pathname, dirnames, filenames in os.walk(path):
        for filename in ( os.path.join(pathname, x) for x in filenames ):
            stat = os.stat(filename)
            yield filename[len(path):], stat.st_mtime, stat.st_size

sys.exit(tuple(getStats(sys.argv[1])) != tuple(getStats(sys.argv[2])))
0

Based on Chris Down's script, this script is a little more "visual". Calling it with two arguments folder1 and folder2, it walks the first folder and for each file searches a corresponding file in the second folder. If it is found, the relative path is printed in green, if they have different modified time or size, it is printed in yellow, and if it is not found then it is printed in red.

#!/usr/bin/env python

import os
import sys
from termcolor import colored

def compare_filestats(file1,file2):
    """
    Compares modified time and size between two files.
    Return:
        -1 if file1 or file2 does not exist
         0 if they exist and compare equal
         1 if they have different modified time, but same size
         2 if they have different size, but same modified time
         3 if they have different size, and different modified time
    """

    if not os.path.exists(file1) or not os.path.exists(file2):
        return -1

    stat1 = os.stat(file1)
    stat2 = os.stat(file2)

    return (stat1.st_mtime != stat2.st_mtime) \
        + 2*(stat1.st_size != stat2.st_size)

def compare_folders(folder1,folder2):
    """
    folder1: serves as reference and will be walked through
    folder2: serves as target and will be querried for each file in folder1

    Prints colored status for each file in folder1:
        missing: file was not found in folder2 
        mtime  : modified time is different
        size   : filesize is different
        ok     : found with same filestats
    """
    for dirpath, dirnames, filenames in os.walk(folder1):
        for file1 in ( os.path.join(dirpath, x) for x in filenames ):
            relpath = file1[len(folder1):]
            file2 = os.path.join( folder2, relpath )
            comp = compare_filestats(file1,file2)

            if comp < 0:
                status = colored('[missing]','red')
            elif comp == 1:
                status = colored('[mtime  ]','yellow')
            elif comp >= 2:
                status = colored('[size   ]','yellow')
            else:
                status = colored('[ok     ]','green')

            print status, relpath

if __name__ == '__main__':
    compare_folders(sys.argv[1],sys.argv[2])

Note that this is not sufficient to decide whether the two folders are the same, you would need to run it both ways to make sure. In practice if you just want to know whether the folders are the same, then Chris' script is better. If you want to know what's missing or different from one folder to another, then my script will tell you.

NOTE: you will need termcolor installed, pip install termcolor.

0

If you'd like to compare only a structure and some basic info about files, you can try something like this:

diff <(cd $DIR1 && ls -laR) <(cd $DIR2 && ls -laR)

I didn't test it, so any edits are welcome :)

  • 2
    This won't work as the directory names themselves will also be in the results. – Chris Down Dec 24 '12 at 19:05
  • what if we will exclude first column with directory names? like <(ls -laR | awk '{$1=""; print}') – Volodymyr Dec 24 '12 at 19:23
  • Not all lines are directory names, so that won't work properly. – Chris Down Dec 24 '12 at 20:19
  • Take advantage of the fact that each <() has its own environment. Edited. – a CVn Dec 11 '16 at 12:43
0

If you only need to know if files from two file system branch are different (without look inside files) you can do something like this:

find /opt/branch1 -type f | sort | xargs -i md5sum {} >/tmp/branch1;
find /opt/branch2 -type f | sort | xargs -i md5sum {} >/tmp/branch2;
diff /tmp/branch1 /tmp/branch2;

HTH

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.