Delete file when there is no difference

Question

I'm completely new to bash. I have a requirement that needs to do the following:

• Iterate through a directory A's and directory B's folders with the same name
• Find two files that have the same name and compare them (im using diff <(file1) <(sort file2) to compare the files)
• If there no differences delete the file in directory A
• If there are differences ignore and process the next matching pair of files
• check the next folder from each directory and repeat the process until all matching folders have been checked.

So for example in Directory A I have folderA that has 2 files (file1 and file2) In directory B I have folderA that has 3 files (file1 and file2 and file3)

• File1 in both directories are the same - Delete from directory A
• file2 there are differences - keep in both directories
• file3 do nothing - keep in directory B

The files that I'm using are xml files. The ordering of tags sometimes differ in the files but the content would be exactly the same, unless there are additions which I'd want to keep the file. I don't necessarily care if the ordering of the tags are different I just want to make sure that all the content are the same or different. Hope that provides more clarity.

Any help would be much appreciated.

UPDATE:

So I've managed to get this far but when running the script the out put in the console is blank. It should list the files that have been found to be the same and remove them, where am I going wrong?

            declare -a my_array
            shopt -s globstar
            cd /mnt/c/filediff/validation/applications/

            for file in **; do
                if [ -d "$file" ]; then 
                    echo "$file is a directory, skipping."; 
                else
                    fileName=${file#*/}
                    if [[ -e /mnt/c/filediff/package/"$fileName" ]]; then
                        echo diff -q <(sort "$file") <(sort /mnt/c/filediff/package/"$fileName") && 
            my_array=("${my_array[@]}" "$fileName")
                        #rm /mnt/c/filediff/package/"$fileName" 
                    fi
                fi
            done
            echo -e '\nRemoved the following files -----------------------------------'
            for item in "${my_array[@]}"
            do
                echo "ITEM: *** $item ***"
            done

Answer 1

You can do something like this:

for file in /path/to/dirA/*; do
    fileName=${file##*/}
    diff -q <(sort "$file") <(sort /path/to/dirB/"$fileName") && 
    rm /path/to/dirB/"$fileName"
done

That will iterate over all files in dirA, saving each as $file. Note that $file will include the path, so it will be /path/to/dirA/file1 and not just file1. This is why we need to get the file name, which we do by removing everything before the last slash (fileName=${file##*/}). Then, we compare the file silently to the file of the same name in directory B and, if they are identical, so if the diff exits successfully, we remove the file from directory B. The && means "run the next command only if this one is successful" so the rm will only run when the files are identical.

To make it recursive, assuming you are using bash, use:

shopt -s globstar
cd /path/to/dirA/
for file in **; do
    fileName=${file#*/}
    echo diff -q <(sort "$file") <(sort /path/to/dirB/"$fileName") && 
    rm /path/to/dirB/"$fileName"
done

Or, a little more sophisticated, skipping directories and non-existent files:

shopt -s globstar
cd /path/to/dirA/
for file in **; do
    if [ -d "$file" ]; then 
        echo "$file is a directory, skipping."; 
    else
        fileName=${file#*/}
        if [[ -e /path/to/dirB/"$fileName" ]]; then
            echo diff -q <(sort "$file") <(sort /path/to/dirB/"$fileName") && 
            rm /path/to/dirB/"$fileName"
        fi
    fi
done

Answer 2

Use a tool such as fdupes, which is especially designed to do this by calculating an md5 hash for each file. I will not give you the specifics here, because there are some caveats, such as loss of data if you specify the same directory twice. For example, if you specify /home/ and ~/ at the same time, the files in your home directory will be listed as their own duplicates. If you'd set fdupes to automatically delete duplicates, you'll end up deleting all the files in your home directory!

Probably the most useful thing you can do, is type man fdupes at your CLI, which will show you the manual with all the options, so you can find for yourself what the best way of using it is, and which options you would want to use or not. The caveats are not something to be scared of, but something to be aware of, and this is probably the best way to train yourself to use such tools, because you'll understand better what it does. You won't get that advantage if you're "blindly" copy-pasting the suggestions given in the answers.

If you don't have the tool available, your distribution will most likely have it available in the repository, as it is the command line tool of choice for many.

Alternatively, you may like to consider fslint, which apparently is a useful GUI-based tool. Never tried it myself, so can't say much about it.