case-insensitive search of duplicate file-names

Question

I there a way to find all files in a directory with duplicate filenames, regardless of the casing (upper-case and/or lower-case)?

Answer 1

If you have GNU utilities (or at least a set that can deal with zero-terminated lines) available, another answer has a great method:

find . -maxdepth 1 -print0 | sort -z | uniq -diz

Note: the output will have zero-terminated strings; the tool you use to further process it should be able to handle that.

In the absence of tools that deal with zero-terminated lines, or if you want to make sure your code works in environments where such tools are not available, you need a small script:

#!/bin/sh
for f in *; do
  find . -maxdepth 1 -iname ./"$f" -exec echo \; | wc -l | while read count; do
    [ $count -gt 1 ] && echo $f
  done
done

What is this madness? See this answer for an explanation of the techniques that make this safe for crazy filenames.

Answer 2

Sort the list of file names in a case-insensitive way and print duplicates. sort has an option for case-insensitive sorting. So does GNU uniq, but not other implementations, and all you can do with uniq is print every element in a set of duplicates except the first that's encountered. With GNU tools, assuming that no file name contains a newline, there's an easy way to print all the elements but one in each set of duplicates:

for x in *; do printf "%s\n" "$x"; done |
sort -f |
uniq -id

Portably, to print all elements in each set of duplicates, assuming that no file name contains a newline:

for x in *; do printf "%s\n" "$x"; done |
sort -f |
awk '
    tolower($0) == tolower(prev) {
        print prev;
        while (tolower($0) == tolower(prev)) {print; getline}
    }
    1 { prev = $0 }'

If you need to accommodate file names containing newlines, go for Perl or Python. Note that you may need to tweak the output, or better do your further processing in the same language, as the sample code below uses newlines to separate names in its own output.

perl -e '
    foreach (glob("*")) {push @{$f{lc($_)}}, $_}
    foreach (keys %f) {@names = @{$f{$_}}; if (@names > 1) {print "$_\n" foreach @names}}
'

Here's a pure zsh solution. It's a bit verbose, as there's no built-in way to keep the duplicate elements in an array or glob result.

a=(*)(N); a=("${(@io)a}")
[[ $#a -le 1 ]] ||
for i in {2..$#a}; do
  if [[ ${(L)a[$i]} == ${(L)a[$((i-1))]} ]]; then
    [[ ${(L)a[$i-2]} == ${(L)a[$((i-1))]} ]] || print -r $a[$((i-1))]
    print -r $a[$i]
  fi
done

Answer 3

There are many complicated answers above, this seems simpler and quicker than all of them:

find . -maxdepth 1 | sort -f | uniq -di

If you want to find duplicate file names in subdirectories then you need to compare just the file name, not the whole path:

find . -maxdepth 2 -printf "%f\n" | sort -f | uniq -di

Edit: Shawn J. Goff has pointed out that this will fail if you have filenames with newline characters. If you're using GNU utilities, you can make these work too:

find . -maxdepth 1 -print0 | sort -fz | uniq -diz

The -print0 (for find) and -z option (for sort and uniq) cause them to work on NUL-terminated strings, instead of newline terminated strings. Since file names can not contain NUL, this works for all file names.

Answer 4

Without GNU find:

LANG=en_US ls | tr '[A-Z]' '[a-z]' | uniq -c | awk '$1 >= 2 {print $2}'

Answer 5

I finally managed it this way:

find . | tr '[:upper:]' '[:lower:]' | sort | uniq -d

I used find instead of ls cause I needed the full path (a lot of subdirectories) included. I did not find how to do this with ls.