How to find and delete duplicate files within the same directory?

Question

I want to find duplicate files, within a directory, and then delete all but one, to reclaim space. How do I achieve this using a shell script?

For example:

pwd
folder

Files in it are:

log.bkp
log
extract.bkp
extract

I need to compare log.bkp with all the other files and if a duplicate file is found (by it's content), I need to delete it. Similarly, file 'log' has to be checked with all other files, that follow, and so on.

So far, I have written this, But it's not giving desired result.

#!/usr/bin/env ksh
count=`ls -ltrh /folder | grep '^-'|wc -l`
for i in `/folder/*`
do
   for (( j=i+1; j<=count; j++ ))
   do
      echo "Current two files are $i and $j"
      sdiff -s $i  $j
      if [ `echo $?` -eq  0 ]
      then
         echo "Contents of $i and $j are same"
       fi
    done
 done

Answer 1

If you're happy to simply use a command line tool, and not have to create a shell script, the fdupes program is available on most distros to do this.

There's also the GUI based fslint tool that has the same functionality.

Answer 2

This solution will find duplicates in O(n) time. Each file has a checksum generated for it, and each file in turn is compared to the set of known checksums via an associative array.

#!/bin/bash
#
# Usage:  ./delete-duplicates.sh  [<files...>]
#
declare -A filecksums

# No args, use files in current directory
test 0 -eq $# && set -- *

for file in "$@"
do
    # Files only (also no symlinks)
    [[ -f "$file" ]] && [[ ! -h "$file" ]] || continue

    # Generate the checksum
    cksum=$(cksum <"$file" | tr ' ' _)

    # Have we already got this one?
    if [[ -n "${filecksums[$cksum]}" ]] && [[ "${filecksums[$cksum]}" != "$file" ]]
    then
        echo "Found '$file' is a duplicate of '${filecksums[$cksum]}'" >&2
        echo rm -f "$file"
    else
        filecksums[$cksum]="$file"
    fi
done

If you don't specify any files (or wildcards) on the command line it will use the set of files in the current directory. It will compare files in multiple directories but it is not written to recurse into directories themselves.

The "first" file in the set is always considered the definitive version. No consideration is taken of file times, permissions or ownerships. Only the content is considered.

Remove the echo from the rm -f "$file" line when you're sure it does what you want. Note that if you were to replace that line with ln -f "${filecksums[$cksum]}" "$file" you could hard-link the content. Same saving in disk space but you wouldn't lose the file names.

Answer 3

The main issue in your script seems to be that i takes the actual file names as values, while j is just a number. Taking the names to an array and using both i and j as indices should work:

files=(*)
count=${#files[@]}
for (( i=0 ; i < count ;i++ )); do 
    for (( j=i+1 ; j < count ; j++ )); do
        if diff -q "${files[i]}" "${files[j]}"  >/dev/null ; then
            echo "${files[i]} and ${files[j]} are the same"
        fi
    done
done

(Seems to work with Bash and the ksh/ksh93 Debian has.)

The assignment a=(this that) would initialize the array a with the two elements this and that (with indices 0 and 1). Wordsplitting and globbing works as usual, so files=(*) initializes files with the names of all files in the current directory (except dotfiles). "${files[@]}" would expand to all elements of the array, and the hash sign asks for a length, so ${#files[@]} is the number of elements in the array. (Note that ${files} would be the first element of the array, and ${#files} is the length of the first element, not the array!)

for i in `/folder/*`

The backticks here are surely a typo? You'd be running the first file as a command and giving the rest as arguments to it.

Answer 4

There are tools that do this, and do it more efficiently. Your solution when it is working is O(n²) that is the time it takes to run is proportional to n² where n is the size of the problem in total bytes in files. The best algorithm could do this in close to O(n). (A am discussing big-O notation, a way to summarise how efficient an algorithm is.)

First you would create a hash of each file, and only compare these: this saves a lot of time if you have a lot of large files that are almost the same.

Second you would use short cut methods: If files have different sizes, then they are not the same. Unless there is another file of same size, don't even open it.

Answer 5

By the way, using checksum or hash is a good idea. My script doesn't use it. But if files are small and amount of files are not big (like 10-20 files), this script will work quite fast. If you are have 100 files and more, 1000 lines in every file, than time will be more than 10 second.

Usage: ./duplicate_removing.sh files/*

#!/bin/bash

for target_file in "$@"; do
    shift
    for candidate_file in "$@"; do
        compare=$(diff -q "$target_file" "$candidate_file")
        if [ -z "$compare" ]; then
            echo the "$target_file" is a copy "$candidate_file"
            echo rm -v "$candidate_file"
        fi
    done
done

Testing

Create random files: ./creating_random_files.sh

#!/bin/bash

file_amount=10
files_dir="files"

mkdir -p "$files_dir"

while ((file_amount)); do
    content=$(shuf -i 1-1000)
    echo "$RANDOM" "$content" | tee "${files_dir}/${file_amount}".txt{,.copied} > /dev/null
    ((file_amount--))
done

Run ./duplicate_removing.sh files/* and get output

the files/10.txt is a copy files/10.txt.copied
rm -v files/10.txt.copied
the files/1.txt is a copy files/1.txt.copied
rm -v files/1.txt.copied
the files/2.txt is a copy files/2.txt.copied
rm -v files/2.txt.copied
the files/3.txt is a copy files/3.txt.copied
rm -v files/3.txt.copied
the files/4.txt is a copy files/4.txt.copied
rm -v files/4.txt.copied
the files/5.txt is a copy files/5.txt.copied
rm -v files/5.txt.copied
the files/6.txt is a copy files/6.txt.copied
rm -v files/6.txt.copied
the files/7.txt is a copy files/7.txt.copied
rm -v files/7.txt.copied
the files/8.txt is a copy files/8.txt.copied
rm -v files/8.txt.copied
the files/9.txt is a copy files/9.txt.copied
rm -v files/9.txt.copied

Answer 6

You can use finddup for this. Read this!!! http://manpages.ubuntu.com/manpages/xenial/man1/finddup.1.html