How to prepare files for rsync on a case insensitive filesystem?

Question

I am transferring a large number of files on a HFS+ filesystem.

The files are currently on ext2 partitions.

I have conflicts due to case insensitivity of the target partition (HFS+).

I want to identify the files that have duplicates filenames once they are in lower case, and delete them if they are actually duplicates.

I also found that I will have duplicate folder names if I convert everyhing to lower case. Basically these hard drives contain years of unsorted data, and I happen to have this problem with folder names too.

Does this seem reasonable:

find . -type f | while read f; do echo $f:l; done | sort | uniq -d 

$f:l is ZSH for convert to lower case.

Now I want to keep only one instance of each file that have duplicates. How to do this efficiently ?

I do not want to find files with duplicate content, unless they have the same lower case filename. I will deal with duplicates later.

Answer 1

The second step in your pipeline is slightly broken (it mangles backslashes and leading and trailing whitespace) and is a complicated way of doing this. Use tr to convert to lowercase. You shouldn't limit the search to files: directories can collide too.

find . | tr '[:upper:]' '[:lower:]' | LC_ALL=C sort | LC_ALL=C uniq -d

Note that this only works if file names don't contain newlines. Under Linux, switch to null bytes as the separator to cope with newlines.

find . -print0 | tr '[:upper:]' '[:lower:]' | LC_ALL=C sort -z | LC_ALL=C uniq -dz

This prints the lowercase versions of file names, which isn't really conducive to doing something about the files.

If you're using zsh, forget about find: zsh has everything you need built in.

setopt extended_glob
for x in **/*; do
  conflicts=($x:h/(#i)$x:t)
  if (($#conflicts > 1)); then
    ## Are all the files identical regular files?
    h=()
    for c in $conflicts; do 
      if [[ -f $c ]]; then
        h+=(${$(md5sum <$c)%% *})
      else
        h=(not regular)
        break
      fi
    done
    if (( ${#${(@u)h}} == 1 )); then
      # Identical regular files, keep only one
      rm -- ${conflicts[1,-2]}
    else
      echo >&2 "Conflicting files:"
      printf >&2 '    %s\n' $conflicts
    fi
  fi
done

Answer 2

I'm working on solution using awk, for the duplicate filenames only, which does not compare the content.

Here the awk file dups.awk

#!/usr/bin/awk -f
{
lc=tolower($0);
count[lc] = count[lc]+1;
tab[lc] = tab[lc] "*" $0;}
END {for (t in tab)
  if (count[t]>1) {
   split(tab[t],sp,"*");
   r=1;sep="# ";
   for (fn in sp) 
      if (length(sp[fn])) 
           {
            print  sep "rm '" sp[fn] "'";
            if (r==1) {r=0; sep="  ";}
            }
   print ""; }
}

I'm calling it like this:

#!/bin/zsh
find $1 -type f | dups.awk

There is one flaw: it won't work with file names with a star in it.

Here in action:

ks% md5sum test/*                               
e342e6ab6ae71954a772409f23390fa4  test/file1
e342e6ab6ae71954a772409f23390fa4  test/File1
e342e6ab6ae71954a772409f23390fa4  test/file2

ks% ./dupsAwk.sh test               
# rm "test/File1"
  rm "test/file1"

Answer 3

Here's a solution using Perl's File::Find instead of trying to work around shell intricacies:

#!/usr/bin/env perl

use strict;
use warnings;
use File::Find;
use Digest::MD5 qw(md5); # To find duplicates

my %lower_case_files_found;
find(
      sub{
          -f or return; # Skip non-files
          push @{$lower_case_files_found{+lc}},$File::Find::name;
      },
      '.'
);
for my $lower_case_name (sort keys %lower_case_files_found){
    my $number_of_files = scalar @{$lower_case_files_found{$lower_case_name}};
    if($number_of_files > 1){
           my %digests_seen;
           for my $file (@{$lower_case_files_found{$lower_case_name}}){
               open my $fh,'<',$file or die "Failed to open $file: $!\n";
               my $file_content = do {local $/;<$fh>};
               my $digest = md5($file_content);
               push @{$digests_seen{$digest}},$file;
           }
           for my $digest (sort keys %digests_seen){
               my $num_of_files = scalar @{$digests_seen{$digest}};
               if ($num_of_files > 1){
                   print "Duplicates: \n";
                   print "[$_]\n" for @{$digests_seen{$digest}}
               }
           }
    }
}

This uses an MD5 sum to determine duplicate files and prints lists of the dupes it found. Each file name is enclosed in [] to help you visually determine file names containing a newline. I deliberately didn't add code to delete any files as this code is completely untested. I leave it up to you to do what you want with the resulting list.

Expect high memory and CPU usage if your files are large in size: the above script loads each file into memory and performs an MD5 sum on its whole contents.

Answer 4

find . -type f |sort |tee f1 |uniq -i |comm -3 - f1

Will give you a list of files to delete or ignore, which you could pipe into an ignore-list for rsync

24hrs later:

In Answer to your comment "It's impractical, I need another find", just pipe the results into something that will do a your rename mangle. eg whole solution on one command line, but less readable.

find . -type f |sort |tee f1 |uniq -i |comm -3 - f1|(n=0;while read a ;do  n=$((${n}+1));echo mv ${a} `echo ${a}|tr \[:upper:\] \[:lower:\]`_renamed_${n};done)