Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to generate a list of files that have:

  • Same name
  • Different content

in a directory (including all children directories and content).

How to do? Bash, perl, anything is fine.

So, two files with the same name and same content should not show up.

share|improve this question
Suppose there are three files with the same name and two of the three are identical? – Kyle Jones Mar 7 '12 at 5:09
@KyleJones: "three files with the same name and two of the three are identical" > Then this filename should be added to the list – Nicolas Raoul Mar 7 '12 at 6:01
up vote 7 down vote accepted

Update: fixed a typo in the script: changed print $NF to print $3; also tidied things up, and added some comments.

Assuming file names do not contain \n, the following prints out a sorted list which breaks (as in: section control breaks) at unique file name, unique md5sum, and shows the corresponding group of file paths.


# Choose which script to use for the final awk step 

# Print all duplicated file names, even when md5sum is the same 
out_all='{ if( p1 != $1 ) { print nl $1; print I $2 }
      else if( p2 != $2 ) { print I $2 }
      print I I $3; p1=$1; p2=$2; nl="\n" }
   END { printf nl}'

# Print only duplicated file names which have multiple md5sums.
out_only='{ if( p1 != $1 ) { if( multi ) { print pend }
                             multi=0; pend=$1 "\n" I $2 "\n" }
       else if( p2 != $2 ) { multi++; pend=pend I $2 "\n" } 
       pend=pend I I $3 "\n"; p1=$1; p2=$2 } 
   END { if( multi ) print pend }'

# The main pipeline 
find "${1:-.}" -type f -name '*' |  # awk for duplicate names
awk -F/ '{ if( name[$NF] ) { dname[$NF]++ }
           name[$NF]=name[$NF] $0 "\n" } 
     END { for( d in dname ) { printf name[d] } 
   }' |                             # standard md5sum output 
xargs -d'\n' md5sum |               # " "==text, "*"==binary
sed 's/ [ *]/\x00/' |               # prefix with file name  
awk -F/ '{ print $3 "\x00" $0 }' |  # sort by name. md5sum, path 
sort |                              # awk to print result
awk -F"\x00" -v"I=   " "${!out_script}"

Output showing only file names with multiple md5s


Output showing all files with the same name.


fi    le.html
      ./test/dir.svn/dir.show/fi    le.html
      ./test/dir.svn/dir.svn/fi    le.html



share|improve this answer

Here's a Perl script. Run it in the directory at the top of the tree you want to search. The script depends on find and md5, but the latter can be replaced with sha1, sum or any other file hashing program that accepts input on stdin and outputs a hash on stdout.

use strict;

my %files;
my %nfiles;
my $HASHER = 'md5';

    for my $x (@_) {
        print "$x\n";

open FINDOUTPUT, "find . -type f -print|" or die "find";

while (defined (my $line = <FINDOUTPUT>)) {
    chomp $line;
    my @segments = split /\//, $line;
    my $shortname = pop @segments;
    push @{ $files{$shortname} }, $line;

for my $shortname (keys %files) {
    if ($nfiles{$shortname} < 2) {
        print_array @{ $files{$shortname} };
    my %nhashes;
    my %revhashes;
    for my $file (@{ $files{$shortname} }) {
        my $hash = `$HASHER < $file`;
        $revhashes{$hash} = $file;
    for my $hash (keys %nhashes) {
        if ($nhashes{$hash} < 2) {
            my $file = $revhashes{$hash};
            print "$file\n";
share|improve this answer

finddup this tool can also help you in listing out the files with same names or content..

share|improve this answer

For those who want to see only a list of filenames, here is the relevant part of Peter.O's answer:

find "${1:-.}" -type f -name '*' | 
awk -F/ '{ if( name[$NF] ) { dname[$NF]++ }
       name[$NF]=name[$NF] $0 "\n" } 
 END { for( d in dname ) { printf name[d] "\n" } 


I don't need md5sums because I use fslint-gui before the script to clear all duplicates.

share|improve this answer
on my mac this shows the duplicate files same name same content – nightograph Mar 25 '15 at 3:25

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.