0

I have a folder with a lot of subfolders and files (let's call it 'orig'), and another folder where a lot of the same files were placed as backups in a different folder structure (let's call it 'backup'). I would like to list all of the files in 'backup' that do not yet exist in 'orig', so that I can put them in the correct subvolder in 'orig' and delete 'backup'. A comparison by file name and size would be sufficient.

Similar questions have been asked before, and the recommended solution is diff -qr orig/ backup/. However, when I tried this, I found that the recursion did not work, as the command listed folders that occured in one but not the other, but not the files. Here is an example. First I create two folders, each with one sub-folder and one file, where each sub-folder and file has a different content:

$ mkdir orig
$ mkdir backup
$ mkdir orig/1
$ mkdir backup/2
$ echo 'blah' > orig/1/test.txt
$ ls orig/1
test.txt
$ echo 'blah1' > backup/2/test1.txt

Now I compare them using the '-qr' option:

$ diff -qr orig/ backup/
Only in orig/: 1
Only in backup/: 2

If I descend into the subfolder in one of them, the file in that folder is found:

$ diff -r orig/1 backup/
Only in backup/: 2
Only in orig/1: test.txt

I actually get the same behaviour with or without '-qr'. Is this a bug or am I misunderstanding diff? I am using diff (GNU diffutils) 3.7.

Any other suggestions for how to solve my problem would be appreciated (e.g. a simple python script).

3
  • Use hardlink (Read man hardlink) to link identical files in the two directory trees. find -links 1 old_dir will find unique files. Check the hardlinks -n log for identical files within old_dir. Hint: grep "old_dir.*old_dir" the log.
    – waltinator
    Feb 15, 2021 at 20:40
  • Thanks for the idea about hardlink! This is amazing. I guess that for the above problem, this would be hardlink orig backup, followed by find backup -links 1, where the latter would return a list of files that have only one hardlink, i.e. only exist in backup. Except that any files that are repeated inside backup would also have more than one hardlink, even if they do not exist in original, right? Feb 15, 2021 at 21:35
  • "Files repeated inside backup" are problematic, which is why I suggested hardlink --verbose --dry-run orig backup |& grep -B3 -A3. 'backup .* -> .*backup', to show files in backup that will be linked to files in backup. Check the surrounding lines for links to orig, but I guess I wasn't clear, sorry.
    – waltinator
    Feb 16, 2021 at 0:13

1 Answer 1

1

I found a way how to solve my problem using python:

import filecmp
import os.path

def find_unique_files(path_orig, path_duplicates, ommit_in_orig=[]):
    """
    Crawls all subfolders of path_duplicates and
    returns list of files (incl. paths) that occur
    in path_duplicates but no-where in path_orig,
    except for folders listed in ommit_in_orig. 
    Do not forget to add trailing '/' at the end of paths.
    
    Arguments:
    path_orig -- string
    path_duplicates -- string
    ommit_in_orig -- list of strings
    
    Returns:
    list of strings indicating paths to files.
    
    Example:
        find_unique_files('/home/user/project/', 
                          '/home/user/project/backups/',
                          ommit_in_orig=['/home/user/project/backups/',
                                         '/home/user/project/temp/'])
    """
    unique_files = []
    for folder, subfolders, files in os.walk(path_duplicates):
        print(folder, end='\r')

        for file in files:
            unique = 1
            filepath = os.path.join(folder,file)
            for folder1, subFolders1, files1 in os.walk(path_orig):
                # Check if folder1 is a subfolder of ommit_in_orig
                ommit = [s for s in ommit_in_orig if s.lower() in folder.lower()]
                if len(ommit) == 0:
                    if file in files1:
                        filepath1 = os.path.join(folder1, file)
                        if filecmp.cmp(filepath,filepath1):
                            # File is identical
                            unique = 0
            if unique == 1:
                unique_files.append(filepath)
    return unique_files
path_orig = 'orig/'
path_duplicates = 'backup/'
find_unique_files(path_orig, path_duplicates)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .