I am transferring a large number of files on a HFS+ filesystem.

The files are currently on ext2 partitions.

I have conflicts due to case insensitivity of the target partition (HFS+).

I want to identify the files that have duplicates filenames once they are in lower case, and delete them if they are actually duplicates.

I also found that I will have duplicate folder names if I convert everyhing to lower case. Basically these hard drives contain years of unsorted data, and I happen to have this problem with folder names too.

Does this seem reasonable:

find . -type f | while read f; do echo $f:l; done | sort | uniq -d 

$f:l is ZSH for convert to lower case.

Now I want to keep only one instance of each file that have duplicates. How to do this efficiently ?

I do not want to find files with duplicate content, unless they have the same lower case filename. I will deal with duplicates later.

  • If by $f:l you mean to convert $f into lowercase, you should note that the POSIX way of doing that is ${f,,*}. Your code will break if a file name contains a newline. You should also note that it will not produce correct paths to the duplicate files because it will lowercase all the path components. – Joseph R. Nov 6 '13 at 11:17
  • OK. This is why I'm asking the question.. – alecail Nov 6 '13 at 11:20
  • Would a Perl solution be acceptable? – Joseph R. Nov 6 '13 at 11:41
  • @JosephR. Absolutely. Anything that runs on *nix is acceptable. The zsh is misleading, maybe I should remove it. – alecail Nov 6 '13 at 11:50

The second step in your pipeline is slightly broken (it mangles backslashes and leading and trailing whitespace) and is a complicated way of doing this. Use tr to convert to lowercase. You shouldn't limit the search to files: directories can collide too.

find . | tr '[:upper:]' '[:lower:]' | LC_ALL=C sort | LC_ALL=C uniq -d

Note that this only works if file names don't contain newlines. Under Linux, switch to null bytes as the separator to cope with newlines.

find . -print0 | tr '[:upper:]' '[:lower:]' | LC_ALL=C sort -z | LC_ALL=C uniq -dz

This prints the lowercase versions of file names, which isn't really conducive to doing something about the files.

If you're using zsh, forget about find: zsh has everything you need built in.

setopt extended_glob
for x in **/*; do
  if (($#conflicts > 1)); then
    ## Are all the files identical regular files?
    for c in $conflicts; do 
      if [[ -f $c ]]; then
        h+=(${$(md5sum <$c)%% *})
        h=(not regular)
    if (( ${#${(@u)h}} == 1 )); then
      # Identical regular files, keep only one
      rm -- ${conflicts[1,-2]}
      echo >&2 "Conflicting files:"
      printf >&2 '    %s\n' $conflicts
  • @AntoineLecaille No. It's a globbing flag. – Gilles Nov 7 '13 at 7:43
  • @AntoineLecaille Ah, I forgot: you need setopt extended_glob (like many other useful globbing features). – Gilles Nov 7 '13 at 13:38
  • rm -- ${conflicts[1,-2]} is dead code. The condition (( ${#${(@u)h}} == 1 )) is wrong. I'm debugging (and learning zsh at the same time).. – alecail Nov 7 '13 at 13:49
  • @AntoineLecaille It's supposed to check if there is a unique element after conversion to lowercase. Ah, the conversion to lowercase is missing! Make that ((${#${(@u)h:l}} == 1)) – Gilles Nov 7 '13 at 13:55
  • I still haven't looked why the condition is not working because I solved my problem with my script. But there is a problem. – alecail Nov 8 '13 at 18:12

I'm working on solution using awk, for the duplicate filenames only, which does not compare the content.

Here the awk file dups.awk

#!/usr/bin/awk -f
count[lc] = count[lc]+1;
tab[lc] = tab[lc] "*" $0;}
END {for (t in tab)
  if (count[t]>1) {
   r=1;sep="# ";
   for (fn in sp) 
      if (length(sp[fn])) 
            print  sep "rm '" sp[fn] "'";
            if (r==1) {r=0; sep="  ";}
   print ""; }

I'm calling it like this:

find $1 -type f | dups.awk

There is one flaw: it won't work with file names with a star in it.

Here in action:

ks% md5sum test/*                               
e342e6ab6ae71954a772409f23390fa4  test/file1
e342e6ab6ae71954a772409f23390fa4  test/File1
e342e6ab6ae71954a772409f23390fa4  test/file2

ks% ./dupsAwk.sh test               
# rm "test/File1"
  rm "test/file1"

Here's a solution using Perl's File::Find instead of trying to work around shell intricacies:

#!/usr/bin/env perl

use strict;
use warnings;
use File::Find;
use Digest::MD5 qw(md5); # To find duplicates

my %lower_case_files_found;
          -f or return; # Skip non-files
          push @{$lower_case_files_found{+lc}},$File::Find::name;
for my $lower_case_name (sort keys %lower_case_files_found){
    my $number_of_files = scalar @{$lower_case_files_found{$lower_case_name}};
    if($number_of_files > 1){
           my %digests_seen;
           for my $file (@{$lower_case_files_found{$lower_case_name}}){
               open my $fh,'<',$file or die "Failed to open $file: $!\n";
               my $file_content = do {local $/;<$fh>};
               my $digest = md5($file_content);
               push @{$digests_seen{$digest}},$file;
           for my $digest (sort keys %digests_seen){
               my $num_of_files = scalar @{$digests_seen{$digest}};
               if ($num_of_files > 1){
                   print "Duplicates: \n";
                   print "[$_]\n" for @{$digests_seen{$digest}}

This uses an MD5 sum to determine duplicate files and prints lists of the dupes it found. Each file name is enclosed in [] to help you visually determine file names containing a newline. I deliberately didn't add code to delete any files as this code is completely untested. I leave it up to you to do what you want with the resulting list.

Expect high memory and CPU usage if your files are large in size: the above script loads each file into memory and performs an MD5 sum on its whole contents.

  • I happen to have two really nasty files: [Cc]hr$(32).ttf .Your script doesn't find them. – alecail Nov 6 '13 at 16:24
  • @AntoineLecaille Please check the updated answer. – Joseph R. Nov 6 '13 at 16:54
find . -type f |sort |tee f1 |uniq -i |comm -3 - f1

Will give you a list of files to delete or ignore, which you could pipe into an ignore-list for rsync

24hrs later:

In Answer to your comment "It's impractical, I need another find", just pipe the results into something that will do a your rename mangle. eg whole solution on one command line, but less readable.

find . -type f |sort |tee f1 |uniq -i |comm -3 - f1|(n=0;while read a ;do  n=$((${n}+1));echo mv ${a} `echo ${a}|tr \[:upper:\] \[:lower:\]`_renamed_${n};done)
  • Note that this assumes that files whose names match case insensitively are actually duplicates, which need not be the case. – Joseph R. Nov 6 '13 at 12:34
  • It gives you a list of files that you will need to do something about, if you don't want to ignore them, you could rename them. – X Tian Nov 6 '13 at 15:08
  • This answers your original question, if you are finding new requirements just adapt the solution I have given. either by quoting the variables and/or adding characters to the translate alphabets. This isn't a discussion board. – X Tian Nov 9 '13 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.