3

Say I have the following directory structure:

root
 |-- dirA
     |-- file.jpg
     |-- file-001.jpg <-- dup
     |-- file2.jpg
     |-- file3.jpg
 |-- dirB
     |-- fileA.jpg
     |-- fileA_ios.jpg <-- dup
     |-- fileB.jpg
     |-- fileC.jpg
 |-- dirC
     |-- fileX.jpg
     |-- fileX_ios.jpg <-- dup
     |-- fileX-001.jpg <-- dup
     |-- fileY.jpg
     |-- fileZ.jpg

So given a root folder, how can I find dups that have identical names (differing only by a suffix) recursively?

The name can be any string, and not necessarily file.... The suffixes can be 001, 002, 003 and so on. But it is safe to assume that there will be a 3-digit numeric pattern and _ios literally (for regex matching).

My linux foo is not very good.

  • Not sure why the other answer was deleted. Seemed to do the job just fine! – Mrchief Nov 27 '14 at 22:41
  • Are you looking for duplicate contents, duplicate filenames (with differing suffixes) or files with the same inodes? – Arcege Nov 28 '14 at 18:39
  • Dup names would suffice for me. While importing pictures, they get named like that depending on how/where they are imported from. Checking size and contents would be a bonus since I can be 100% sure then, but even without that, name matching is acceptable. – Mrchief Nov 28 '14 at 23:35
2

It is a slightly long, but it is a single command-line. This looks at the contents of the files and compares them using a cryptographic hash (md5sum).

find . -type f -exec md5sum {} + | sort | sed 's/  */!/1' | awk -F\| 'BEGIN{first=1}{if($1==lastid){if(first){first=0;print lastid, lastfile}print$1, $2} else first=1; lastid=$1;lastfile=$2}'

As I said, this is a little long...

The find runs md5sum against all files in the current directory tree. Then the output is sortd by the md5 hash. Since whitespace could be in the filenames, the sed changes the first field separator (two spaces) to a vertical pipe (very unlikely to be in a filename).

The last awk command tracks three variables: lastid = the md5 hash from the previous entry, lastfile = the filename from previous entry, and first = lastid was first time seen.

The output includes the hash so you can see which files are duplicates of each other.

This does not indicate if files are hard links (same inode, different name); it will just compare the contents.

Update: corrected based on just basename of file.

find . -type f -print | sed 's!.*/\(.*\)\.[^.]*$!\1|&!' | awk -F\| '{i=indices[$1]++;found[$1,i]=$2}END{for(bname in indices){if(indices[bname]>1){for(i=0;i<indices[bname];i++){print found[bname,i]}}}}'

Here, the find just lists the filenames, the sed takes the basename component of the pathname and creates a two field table with the basename and the full pathname. The awk then creates a table ("found") of the pathnames seen, indexed by the basename and the item number; the "indices" array keeps track of how many of that basename have been seen. The "END" clause then prints out any duplicate basenames found.

  • Appreciate the answer! However, this wasn't able to find a single dup and I think that's because of md5sum. Pictures can have extra info (exif tags) which may or may not matter. A strict md5sum will fail since all the bytes don't line up. Besides, content check IMO should always be the second check after name matching. Name matching will narrow it down to a handful and content match will ensure accuracy. That way, your command will also run faster since its operating on a handful of files. – Mrchief Nov 30 '14 at 16:25
  • See update of the answer. – Arcege Nov 30 '14 at 18:27
  • Still not quite there. It finds same name in different directories, and not identical ones in same directory. – Mrchief Dec 2 '14 at 1:47
0

You may want to consider programs specifically intended to the search of duplicate files rather than relying on the name, e.g. fdupes or fslint.

  • The problem with those is that all the files will come up as dups, since they have identical names (except for suffix). Or maybe I'm missing something here? – Mrchief Nov 27 '14 at 21:46
0

Create directory structure

mkdir dir{A,B,C}
touch dirA/file{,-001,2,3}.jpg
touch dirB/file{A,A_ios,B,C}.jpg
touch dirC/file{X,X_ios,X-001,Y,Z}.jpg

Show multiplicity of duplicate files

find . -name '*.jpg' -type f |sed 's/\(.*\/\(file.\).*\(.jpg\)\)/\2/' |sort |uniq -c|grep -v 1 

Returns

2 fileA
3 fileX

  • This is not what I'm after. There are 100s of directories and the name can be anything (not necessarily file...). – Mrchief Nov 28 '14 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.