3

How is it possible to list duplicate file names on a Linux system

  • ignoring the case
  • including all subdirectories

Files should not be compared by their content but only by their names. The output should be a list of file names including the path, so that one can run further commands on these files.

Lets assume we have

ls -1R /tmp/
foo
BAR
barfoo
a/BAr
a/b/bar
c/bAr

The output of the filter/find script should be

/tmp/BAR
/tmp/a/BAr
/tmp/a/b/bar
/tmp/c/bAr
  • What do you mean by Files should not be compared by their content by intention. So, if you have file1 and file2 with exact same contents, do you want them to be listed or not listed? – Ramesh Oct 31 '14 at 16:39
  • may be some find . -print | sed is the right strategy... – Jonas Stein Oct 31 '14 at 16:59
  • 1
    Well, if you are on an ubuntu machine this might help. – Ramesh Oct 31 '14 at 17:00
3
find . -printf "%p %f\n" | sort -f -k2 | uniq -Di -f1

Specify your choice of starting directory for find if you don’t want to start at ..  Add -type f if you want just file names. 

  • The find command produces a list of file (and directory) names, in directory order (i.e., random order, as far as you’re concerned).
  • -printf "%p %f\n" prints the full pathname (relative to .) and the filename. 
  • sort -f is short for sort --ignore-case, i.e., it sorts the filename list in a case-insensitive way
  • -k2 tells it to use the second field as the sort key. 
  • uniq -Di -f1 is short for uniq --all-repeated --ignore-case --skip-fields=1, i.e., it shows (all) the lines of output from find that occur repeatedly, based on case-insensitive comparison of the second field and beyond (i.e., that have the same (case-insensitive) file name).

This should give you the output that you want, except each line will have the filename repeated at the end.  If you want to get rid of that, pipe into sed 's/ .*//'.

A couple of gotchas:

  • If you have directories whose names are the same except for case, and they contain files whose names are the same except for case, e.g.,

    documents/design.doc
    Documents/Design.doc
    

    then these will be listed.

  • If you have files (or directories) whose names contain spaces, tabs, or newlines, this will break.

  • This won't work. Will only match files where the path and the filename both match (except for case). It won't match /tmp/BAR and /tmp/a/BAr. – Omnipresence Oct 31 '14 at 16:57
  • But the question is not (only) about files in the same directory... – Hauke Laging Oct 31 '14 at 17:00
  • Well, that was far from obvious until the OP edited the question, which happened while I was typing my answer. – Scott Oct 31 '14 at 17:09
2

This works for basic file names, but will not work for files with carriage returns and probably a few other edge cases.

FilesNoPath=$(find . -printf "%f\n")
FilesWithPath=$(find .)

oldIFS=$IFS
IFS=$'\n'
for filename in $FilesNoPath;
do
    Matches=$(echo "$FilesWithPath" | grep -i "/$filename$")
    if [ $(echo "$Matches" | wc -l) -gt 1 ]
    then
        echo Found matches:
        echo "$Matches"
        FilesWithPath=$(echo "$FilesWithPath" | grep -vi "/$filename$")
    fi
done
IFS=$oldIFS

You can replace the . in the two find commands with /path/to/your/files if you want the output to show the full path and/or you don't want to execute this from within the parent directory of the matching files.

  • 2
    Don't you need IFS=$'\n' so that spaces in filenames won't break the for loop? – Barmar Oct 31 '14 at 20:20
  • 1
    @Barmar You're absolutely right. Edited. – Omnipresence Nov 3 '14 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.