1

I'm completely new to bash. I have a requirement that needs to do the following:

  • Iterate through a directory A's and directory B's folders with the same name
  • Find two files that have the same name and compare them (im using diff <(file1) <(sort file2) to compare the files)
  • If there no differences delete the file in directory A
  • If there are differences ignore and process the next matching pair of files
  • check the next folder from each directory and repeat the process until all matching folders have been checked.

So for example in Directory A I have folderA that has 2 files (file1 and file2) In directory B I have folderA that has 3 files (file1 and file2 and file3)

  • File1 in both directories are the same - Delete from directory A
  • file2 there are differences - keep in both directories
  • file3 do nothing - keep in directory B

The files that I'm using are xml files. The ordering of tags sometimes differ in the files but the content would be exactly the same, unless there are additions which I'd want to keep the file. I don't necessarily care if the ordering of the tags are different I just want to make sure that all the content are the same or different. Hope that provides more clarity.

Any help would be much appreciated.

UPDATE:

So I've managed to get this far but when running the script the out put in the console is blank. It should list the files that have been found to be the same and remove them, where am I going wrong?

blank

            declare -a my_array
            shopt -s globstar
            cd /mnt/c/filediff/validation/applications/

            for file in **; do
                if [ -d "$file" ]; then 
                    echo "$file is a directory, skipping."; 
                else
                    fileName=${file#*/}
                    if [[ -e /mnt/c/filediff/package/"$fileName" ]]; then
                        echo diff -q <(sort "$file") <(sort /mnt/c/filediff/package/"$fileName") && 
            my_array=("${my_array[@]}" "$fileName")
                        #rm /mnt/c/filediff/package/"$fileName" 
                    fi
                fi
            done
            echo -e '\nRemoved the following files -----------------------------------'
            for item in "${my_array[@]}"
            do
                echo "ITEM: *** $item ***"
            done
  • Could you please add a bit more about your definition of when two files are to be considered the same? Is the ordering of the lines in the files unimportant for the comparison, and is one of the files already sorted? As you may see in comments below, one of the answers depend on comparing the files as they are (with no sorting), and the other is sorting both files before comparing. – Kusalananda Jan 7 at 19:56
  • Hmm. That sounds like you have no files or directories in the target path. What do you get if you just run shopt -s globstar; for f in /mnt/c/filediff/validation/applications/**; do echo "F: $f"; done? Does that list any files? – terdon Jan 13 at 16:38
  • @terdon yeah that works, its listing all my files in that folder – Minz Jan 14 at 8:14
1

You can do something like this:

for file in /path/to/dirA/*; do
    fileName=${file##*/}
    diff -q <(sort "$file") <(sort /path/to/dirB/"$fileName") && 
    rm /path/to/dirB/"$fileName"
done

That will iterate over all files in dirA, saving each as $file. Note that $file will include the path, so it will be /path/to/dirA/file1 and not just file1. This is why we need to get the file name, which we do by removing everything before the last slash (fileName=${file##*/}). Then, we compare the file silently to the file of the same name in directory B and, if they are identical, so if the diff exits successfully, we remove the file from directory B. The && means "run the next command only if this one is successful" so the rm will only run when the files are identical.

To make it recursive, assuming you are using bash, use:

shopt -s globstar
cd /path/to/dirA/
for file in **; do
    fileName=${file#*/}
    echo diff -q <(sort "$file") <(sort /path/to/dirB/"$fileName") && 
    rm /path/to/dirB/"$fileName"
done

Or, a little more sophisticated, skipping directories and non-existent files:

shopt -s globstar
cd /path/to/dirA/
for file in **; do
    if [ -d "$file" ]; then 
        echo "$file is a directory, skipping."; 
    else
        fileName=${file#*/}
        if [[ -e /path/to/dirB/"$fileName" ]]; then
            echo diff -q <(sort "$file") <(sort /path/to/dirB/"$fileName") && 
            rm /path/to/dirB/"$fileName"
        fi
    fi
done
  • Thanks this makes sense, there's just one more thing that I forgot to include, in dirA there are sub-directories, how would I go through each sub-directory? would I need another for loop? – Minz Jan 7 at 9:58
  • @Minz If you need to clarify the question, then please do so in the question and not in comments. – Kusalananda Jan 7 at 10:11
  • @Minz that's a pretty different question. Would there be the same subdirectories in dirB? Or would we need to check all file names from dirA to all files from dirB and its subdirs? Please try to ensure that the question you ask accurately represents your situation. – terdon Jan 7 at 10:15
  • @terdon I've updated the question – Minz Jan 7 at 10:57
  • 1
    I certainly would not use sort or diff to decide whether arbitrary files are identical. A pair of files, one of which is a sorted version of the other, are different precisely by virtue of one of them being sorted. I wouldn't even use diff: both sort and diff work on the assumption that data is line-based text. cmp -s would be reasonable. Personally, I would cksum the files: this has the great advantage that you can use unique checksum output to determine that many copies of a file (even with different names) are identical, whereas diff and com only compare two specified files at a time. – Paul_Pedant Jan 7 at 17:11
0

Use a tool such as fdupes, which is especially designed to do this by calculating an md5 hash for each file. I will not give you the specifics here, because there are some caveats, such as loss of data if you specify the same directory twice. For example, if you specify /home/ and ~/ at the same time, the files in your home directory will be listed as their own duplicates. If you'd set fdupes to automatically delete duplicates, you'll end up deleting all the files in your home directory!

Probably the most useful thing you can do, is type man fdupes at your CLI, which will show you the manual with all the options, so you can find for yourself what the best way of using it is, and which options you would want to use or not. The caveats are not something to be scared of, but something to be aware of, and this is probably the best way to train yourself to use such tools, because you'll understand better what it does. You won't get that advantage if you're "blindly" copy-pasting the suggestions given in the answers.

If you don't have the tool available, your distribution will most likely have it available in the repository, as it is the command line tool of choice for many.

Alternatively, you may like to consider fslint, which apparently is a useful GUI-based tool. Never tried it myself, so can't say much about it.

  • Note that the user has (seems to have) a specific sense in which two files are considered duplicates. If one of the files, sorted, is the same as the other, they are duplicates. – Kusalananda Jan 7 at 19:53
  • 1
    I appreciate what you are saying: the user has chosen a specific method. But at the beginning, the user notes "I'm completely new to bash", then in the 2nd step compares two files of the same name to see if they are the same: any differences in a file with the same name, keep both. In the process, one random file is sorted, the other is not. The only way to get a match is if both are empty files or accidentally end up sorting the right one. From my perspective, this is a learner who found some commands/tools that seem helpful, but may not be the right ones to accomplish the purpose. – Mr. Donutz Jan 7 at 20:04
  • Well, I asked them to comment upon it. I can imagine comparing two files in the way they show being a totally valid way to compare two files when the data in them has been collected by two different methods, or at two different points in time, and the ordering of it does not matter. – Kusalananda Jan 7 at 20:07
  • Probably the only way to find out actively, where I've taken a more passive approach of waiting to see whether I get any comments from the user to feedback on my answer, which happens at times, if considered useful. // Edit: here's an example of it happening just now unix.stackexchange.com/questions/560893/… – Mr. Donutz Jan 7 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.