| |-- a.txt
| |-- b.txt
| `-- c.txt
2 directories, 6 files
$ fdupes -1 A B
fdupes detects duplicates based on file contents. The
-1 flag makes it output the filenames of each set of duplicates on a single line. Here, it detects that the
b.txt files are identical.
You may use
fdupes to delete duplicates:
$ fdupes --delete A B
Set 1 of 1, preserve files [1 - 2, all]: 1
It interactively asks which file to keep (or to keep both). I wrote
1 so the
A/b.txt file was kept while
B/b.txt was deleted.
See the manual for
man fdupes). If it's not installed on your system, then use a package manager to install it. It can also be made to automatically delete files without interactive prompting, but care must be taken when running it in this way. Always make a backup of your data before running a command that may delete files.
fdupes will always keep at least one of the duplicates. If you want to delete all duplicates, then you may be interested in this patched version of
fdupes mentioned in an answer to similar question over at SuperUser: https://superuser.com/a/947770/96962 (I have not tested this).
The reason I suggest using
fdupes rather than parsing the log file that you have is that filenames embedded in a text document are difficult to parse correctly. It may not always be difficult (and is this particular example, it would be easy), but note that Unix allows for both spaces and newlines in the names of files and directories. It is technically possible to have a directory called
with a newline embedded in the name.