4

I am transferring a large number of files on a HFS+ filesystem.

The files are currently on ext2 partitions.

I have conflicts due to case insensitivity of the target partition (HFS+).

I want to identify the files that have duplicates filenames once they are in lower case, and delete them if they are actually duplicates.

I also found that I will have duplicate folder names if I convert everyhing to lower case. Basically these hard drives contain years of unsorted data, and I happen to have this problem with folder names too.

Does this seem reasonable:

find . -type f | while read f; do echo $f:l; done | sort | uniq -d 

$f:l is ZSH for convert to lower case.

Now I want to keep only one instance of each file that have duplicates. How to do this efficiently ?

I do not want to find files with duplicate content, unless they have the same lower case filename. I will deal with duplicates later.

4
  • If by $f:l you mean to convert $f into lowercase, you should note that the POSIX way of doing that is ${f,,*}. Your code will break if a file name contains a newline. You should also note that it will not produce correct paths to the duplicate files because it will lowercase all the path components.
    – Joseph R.
    Nov 6, 2013 at 11:17
  • OK. This is why I'm asking the question..
    – alecail
    Nov 6, 2013 at 11:20
  • Would a Perl solution be acceptable?
    – Joseph R.
    Nov 6, 2013 at 11:41
  • @JosephR. Absolutely. Anything that runs on *nix is acceptable. The zsh is misleading, maybe I should remove it.
    – alecail
    Nov 6, 2013 at 11:50

4 Answers 4

3

The second step in your pipeline is slightly broken (it mangles backslashes and leading and trailing whitespace) and is a complicated way of doing this. Use tr to convert to lowercase. You shouldn't limit the search to files: directories can collide too.

find . | tr '[:upper:]' '[:lower:]' | LC_ALL=C sort | LC_ALL=C uniq -d

Note that this only works if file names don't contain newlines. Under Linux, switch to null bytes as the separator to cope with newlines.

find . -print0 | tr '[:upper:]' '[:lower:]' | LC_ALL=C sort -z | LC_ALL=C uniq -dz

This prints the lowercase versions of file names, which isn't really conducive to doing something about the files.

If you're using zsh, forget about find: zsh has everything you need built in.

setopt extended_glob
for x in **/*; do
  conflicts=($x:h/(#i)$x:t)
  if (($#conflicts > 1)); then
    ## Are all the files identical regular files?
    h=()
    for c in $conflicts; do 
      if [[ -f $c ]]; then
        h+=(${$(md5sum <$c)%% *})
      else
        h=(not regular)
        break
      fi
    done
    if (( ${#${(@u)h}} == 1 )); then
      # Identical regular files, keep only one
      rm -- ${conflicts[1,-2]}
    else
      echo >&2 "Conflicting files:"
      printf >&2 '    %s\n' $conflicts
    fi
  fi
done
5
  • @AntoineLecaille No. It's a globbing flag. Nov 7, 2013 at 7:43
  • @AntoineLecaille Ah, I forgot: you need setopt extended_glob (like many other useful globbing features). Nov 7, 2013 at 13:38
  • rm -- ${conflicts[1,-2]} is dead code. The condition (( ${#${(@u)h}} == 1 )) is wrong. I'm debugging (and learning zsh at the same time)..
    – alecail
    Nov 7, 2013 at 13:49
  • @AntoineLecaille It's supposed to check if there is a unique element after conversion to lowercase. Ah, the conversion to lowercase is missing! Make that ((${#${(@u)h:l}} == 1)) Nov 7, 2013 at 13:55
  • I still haven't looked why the condition is not working because I solved my problem with my script. But there is a problem.
    – alecail
    Nov 8, 2013 at 18:12
1

I'm working on solution using awk, for the duplicate filenames only, which does not compare the content.

Here the awk file dups.awk

#!/usr/bin/awk -f
{
lc=tolower($0);
count[lc] = count[lc]+1;
tab[lc] = tab[lc] "*" $0;}
END {for (t in tab)
  if (count[t]>1) {
   split(tab[t],sp,"*");
   r=1;sep="# ";
   for (fn in sp) 
      if (length(sp[fn])) 
           {
            print  sep "rm '" sp[fn] "'";
            if (r==1) {r=0; sep="  ";}
            }
   print ""; }
}

I'm calling it like this:

#!/bin/zsh
find $1 -type f | dups.awk

There is one flaw: it won't work with file names with a star in it.

Here in action:

ks% md5sum test/*                               
e342e6ab6ae71954a772409f23390fa4  test/file1
e342e6ab6ae71954a772409f23390fa4  test/File1
e342e6ab6ae71954a772409f23390fa4  test/file2

ks% ./dupsAwk.sh test               
# rm "test/File1"
  rm "test/file1"
0

Here's a solution using Perl's File::Find instead of trying to work around shell intricacies:

#!/usr/bin/env perl

use strict;
use warnings;
use File::Find;
use Digest::MD5 qw(md5); # To find duplicates

my %lower_case_files_found;
find(
      sub{
          -f or return; # Skip non-files
          push @{$lower_case_files_found{+lc}},$File::Find::name;
      },
      '.'
);
for my $lower_case_name (sort keys %lower_case_files_found){
    my $number_of_files = scalar @{$lower_case_files_found{$lower_case_name}};
    if($number_of_files > 1){
           my %digests_seen;
           for my $file (@{$lower_case_files_found{$lower_case_name}}){
               open my $fh,'<',$file or die "Failed to open $file: $!\n";
               my $file_content = do {local $/;<$fh>};
               my $digest = md5($file_content);
               push @{$digests_seen{$digest}},$file;
           }
           for my $digest (sort keys %digests_seen){
               my $num_of_files = scalar @{$digests_seen{$digest}};
               if ($num_of_files > 1){
                   print "Duplicates: \n";
                   print "[$_]\n" for @{$digests_seen{$digest}}
               }
           }
    }
}

This uses an MD5 sum to determine duplicate files and prints lists of the dupes it found. Each file name is enclosed in [] to help you visually determine file names containing a newline. I deliberately didn't add code to delete any files as this code is completely untested. I leave it up to you to do what you want with the resulting list.

Expect high memory and CPU usage if your files are large in size: the above script loads each file into memory and performs an MD5 sum on its whole contents.

2
  • I happen to have two really nasty files: [Cc]hr$(32).ttf .Your script doesn't find them.
    – alecail
    Nov 6, 2013 at 16:24
  • @AntoineLecaille Please check the updated answer.
    – Joseph R.
    Nov 6, 2013 at 16:54
0
find . -type f |sort |tee f1 |uniq -i |comm -3 - f1

Will give you a list of files to delete or ignore, which you could pipe into an ignore-list for rsync

24hrs later:

In Answer to your comment "It's impractical, I need another find", just pipe the results into something that will do a your rename mangle. eg whole solution on one command line, but less readable.

find . -type f |sort |tee f1 |uniq -i |comm -3 - f1|(n=0;while read a ;do  n=$((${n}+1));echo mv ${a} `echo ${a}|tr \[:upper:\] \[:lower:\]`_renamed_${n};done)
3
  • Note that this assumes that files whose names match case insensitively are actually duplicates, which need not be the case.
    – Joseph R.
    Nov 6, 2013 at 12:34
  • It gives you a list of files that you will need to do something about, if you don't want to ignore them, you could rename them.
    – X Tian
    Nov 6, 2013 at 15:08
  • This answers your original question, if you are finding new requirements just adapt the solution I have given. either by quoting the variables and/or adding characters to the translate alphabets. This isn't a discussion board.
    – X Tian
    Nov 9, 2013 at 0:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .