I want to find duplicate files, within a directory, and then delete all but one, to reclaim space. How do I achieve this using a shell script?

For example:


Files in it are:


I need to compare log.bkp with all the other files and if a duplicate file is found (by it's content), I need to delete it. Similarly, file 'log' has to be checked with all other files, that follow, and so on.

So far, I have written this, But it's not giving desired result.

#!/usr/bin/env ksh
count=`ls -ltrh /folder | grep '^-'|wc -l`
for i in `/folder/*`
   for (( j=i+1; j<=count; j++ ))
      echo "Current two files are $i and $j"
      sdiff -s $i  $j
      if [ `echo $?` -eq  0 ]
         echo "Contents of $i and $j are same"
  • 2
    note echo $? is equivalent to $?. Also $(command) is preferred to using back ticks. May 28, 2017 at 16:50
  • Here is a one-line script to find duplicate files with subfolders support, it may help others also: md5sum `find -type f` | sort | uniq -D -w 32 (superuser.com/questions/259148/…) It does not automatically delete duplicates but it could be changed if you really want to.
    – baptx
    Dec 30, 2018 at 16:31

6 Answers 6


If you're happy to simply use a command line tool, and not have to create a shell script, the fdupes program is available on most distros to do this.

There's also the GUI based fslint tool that has the same functionality.

  • Both of these utilities are not installed in my systems! May 29, 2017 at 8:11
  • 2
    @Su_scriptingbee If you let us know what your system is, a way to install it could possible be suggested.
    – einonm
    May 29, 2017 at 12:14
  • @Su_scriptingbee you can brew install fdupes if you have brew installed.
    – A.G.
    Jun 1, 2021 at 18:00

This solution will find duplicates in O(n) time. Each file has a checksum generated for it, and each file in turn is compared to the set of known checksums via an associative array.

# Usage:  ./delete-duplicates.sh  [<files...>]
declare -A filecksums

# No args, use files in current directory
test 0 -eq $# && set -- *

for file in "$@"
    # Files only (also no symlinks)
    [[ -f "$file" ]] && [[ ! -h "$file" ]] || continue

    # Generate the checksum
    cksum=$(cksum <"$file" | tr ' ' _)

    # Have we already got this one?
    if [[ -n "${filecksums[$cksum]}" ]] && [[ "${filecksums[$cksum]}" != "$file" ]]
        echo "Found '$file' is a duplicate of '${filecksums[$cksum]}'" >&2
        echo rm -f "$file"

If you don't specify any files (or wildcards) on the command line it will use the set of files in the current directory. It will compare files in multiple directories but it is not written to recurse into directories themselves.

The "first" file in the set is always considered the definitive version. No consideration is taken of file times, permissions or ownerships. Only the content is considered.

Remove the echo from the rm -f "$file" line when you're sure it does what you want. Note that if you were to replace that line with ln -f "${filecksums[$cksum]}" "$file" you could hard-link the content. Same saving in disk space but you wouldn't lose the file names.

  • It will be helpful if you elaborate on "test 0 -eq $# && set -- * " May 29, 2017 at 8:08
  • 1
    @Su_scriptingbee: could be rewritten as : [ 0 -eq $# ] && set -- * . To explain: test 0 -eq "$#" compares (numerically, with -eq) "0" with "$#" (which is the number of arguments passed to the shell script). If 0, there was no arguments passed, and then (&&) it creates an argument list with set -- * (* which will expand to all the files and dirs in the current directory (as the script as no "cd" before that, it will be the directory the person was in when launching the script, or the home-dir of the user starting the script if the script is launched remotely or via something like cron). May 30, 2017 at 9:17
  • @Su_scriptingbee test 0 -eq $# asks if there are any command-line arguments. set -- * sets the command-line arguments to be the set of items (files, directories, etc.) in the current directory. The && allows the second part to run only if the the first part succeeds.
    – roaima
    May 30, 2017 at 9:18

The main issue in your script seems to be that i takes the actual file names as values, while j is just a number. Taking the names to an array and using both i and j as indices should work:

for (( i=0 ; i < count ;i++ )); do 
    for (( j=i+1 ; j < count ; j++ )); do
        if diff -q "${files[i]}" "${files[j]}"  >/dev/null ; then
            echo "${files[i]} and ${files[j]} are the same"

(Seems to work with Bash and the ksh/ksh93 Debian has.)

The assignment a=(this that) would initialize the array a with the two elements this and that (with indices 0 and 1). Wordsplitting and globbing works as usual, so files=(*) initializes files with the names of all files in the current directory (except dotfiles). "${files[@]}" would expand to all elements of the array, and the hash sign asks for a length, so ${#files[@]} is the number of elements in the array. (Note that ${files} would be the first element of the array, and ${#files} is the length of the first element, not the array!)

for i in `/folder/*`

The backticks here are surely a typo? You'd be running the first file as a command and giving the rest as arguments to it.

  • Thank you!! Its giving me the output. Could you pls elaborate on first two lines? May 28, 2017 at 17:25
  • @Su_scriptingbee, elaborated. Bash's manual has a page on arrays if you need a reference. I think it should be similar in ksh, except that bash uses declare in place of ksh's typeset. (I may have missed some differences, though.)
    – ilkkachu
    May 28, 2017 at 17:38

There are tools that do this, and do it more efficiently. Your solution when it is working is O(n²) that is the time it takes to run is proportional to n² where n is the size of the problem in total bytes in files. The best algorithm could do this in close to O(n). (A am discussing big-O notation, a way to summarise how efficient an algorithm is.)

First you would create a hash of each file, and only compare these: this saves a lot of time if you have a lot of large files that are almost the same.

Second you would use short cut methods: If files have different sizes, then they are not the same. Unless there is another file of same size, don't even open it.


By the way, using checksum or hash is a good idea. My script doesn't use it. But if files are small and amount of files are not big (like 10-20 files), this script will work quite fast. If you are have 100 files and more, 1000 lines in every file, than time will be more than 10 second.

Usage: ./duplicate_removing.sh files/*


for target_file in "$@"; do
    for candidate_file in "$@"; do
        compare=$(diff -q "$target_file" "$candidate_file")
        if [ -z "$compare" ]; then
            echo the "$target_file" is a copy "$candidate_file"
            echo rm -v "$candidate_file"


Create random files: ./creating_random_files.sh



mkdir -p "$files_dir"

while ((file_amount)); do
    content=$(shuf -i 1-1000)
    echo "$RANDOM" "$content" | tee "${files_dir}/${file_amount}".txt{,.copied} > /dev/null

Run ./duplicate_removing.sh files/* and get output

the files/10.txt is a copy files/10.txt.copied
rm -v files/10.txt.copied
the files/1.txt is a copy files/1.txt.copied
rm -v files/1.txt.copied
the files/2.txt is a copy files/2.txt.copied
rm -v files/2.txt.copied
the files/3.txt is a copy files/3.txt.copied
rm -v files/3.txt.copied
the files/4.txt is a copy files/4.txt.copied
rm -v files/4.txt.copied
the files/5.txt is a copy files/5.txt.copied
rm -v files/5.txt.copied
the files/6.txt is a copy files/6.txt.copied
rm -v files/6.txt.copied
the files/7.txt is a copy files/7.txt.copied
rm -v files/7.txt.copied
the files/8.txt is a copy files/8.txt.copied
rm -v files/8.txt.copied
the files/9.txt is a copy files/9.txt.copied
rm -v files/9.txt.copied
  • Cool, this is working. I just tested on dir with less than 10 files. May 29, 2017 at 8:05

You can use finddup for this. Read this!!! http://manpages.ubuntu.com/manpages/xenial/man1/finddup.1.html

  • finddup can find duplicates. please read the manual page. As well as you can use fdupes. Dec 1, 2017 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.