5

I have directory that has a number of sub-directories and would like to find any duplicates. The folder structure looks something like this:

└── Top_Dir
    └── Level_1_Dir
        ├── standard_cat
        │   ├── files.txt
        ├── standard_dog
        │   └── files.txt
        └── standard_snake
            └── files.txt
    └── Level_2_Dir
        ├── standard_moon
        │   ├── files.txt
        ├── standard_sun
        │   └── files.txt
        └── standard_cat
            └── files.txt
    └── Level_3_Dir
        ├── standard_man
        │   ├── files.txt
        ├── standard_woman
        │   └── files.txt
        └── standard_moon
            └── files.txt

With the above example I would like to see an output of:

/top_dir/Level_1_Dir/standard_cat
/top_dir/Level_2_Dir/standard_cat
/top_dir/Level_2_Dir/standard_moon
/top_dir/Level_3_Dir/standard_moon

I have been doing some searching on how to get this done via bash and I got nothing. Anyone know a way to do this?

2
0

I had the same problem with my music collection... most tools/scripts were noisy (listing filenames) or did checksums of file contents, which is far too slow...

Special characters, spaces, and symbols made this challenging... the strategy is to MD5sum the sorted file names along with the parent directory, then the script can sort hashes to find duplicates. We must sort children file names, as find does not guarantee file order in two different directories.

Bash Script (Debian 10):

#!/bin/bash

# usage: ./find_duplicates tunes_dir
# output: c547c3bcf85b9c578a1a52dd20665343 - /mnt/tunes/soul brothers/Motherlode
# MD5 is generated from all children filenames + album folder name
# sort list by MD5 then list duplicate (32bit hashes) representing albums
# Album/CD1/... Album/CD2/... will show (3) results if Album is duplicated
# CD1/2 example is indistinguishable from Discography/Album/Song.mp3

if [ $# -eq 0 ]; then
    echo "Please supply tunes directory as first arg"
    exit 1
fi

# Using absolute path of tunes_dir param
find $(readlink -f $1) -type d | while IFS= read -r line
do
    cd "$line"
    children=$(find ./ -type f | sort)
    base=$(basename "$line")
    sum=$(echo $children $base | md5sum)
    echo $sum $line
done | sort -n | uniq -D -w 32

Directory structure:

user@pc:~/test# find . -type d
./super soul brothers
./super soul brothers/Stritch's Brew
./super soul brothers/Fireball!
./super soul brothers/Motherlode
./car_tunes
./car_tunes/Fireball!

Example output:

user@pc:~# ./find_duplicates  test/
07b0f79429663685f4005486af20247a - /root/test/car_tunes/Fireball!
07b0f79429663685f4005486af20247a - /root/test/super soul brothers/Fireball!
0

Using bash release 4 or newer. On macOS, this may be installed via the Homebrew package manager, as the default bash is too old.

# Make glob patterns disappear rather than remain unexpanded
# if the don't match (nullglob).
# Make glob patterns also match hidden names (dotglob).
shopt -s nullglob dotglob

# Create an associative array that hold the number of times
# a directory's name has been seen (the basename of the directory's
# pathname is the key into this array).
declare -A count

# Set the positional parameters ($1, $2, etc.) to the pathnames
# of the directories that we're interested in.
set -- Top_Dir/*/*/

# Loop over out directory paths,
# and count how many times each basename occurs.
for dirpath do
        name=$( basename "$dirpath" )
        count["$name"]=$(( count["$name"] + 1 ))
done

# Loop over the directory paths again, but this time
# output each directory whose basename occurs more than once.
for dirpath do
        name=$( basename "$dirpath" )
        [[ ${count["$name"]} -gt 1 ]] && printf '%s\n' "$dirpath"
done

Testing:

$ tree -F
.
|-- Top_Dir/
|   |-- Level_1_Dir/
|   |   |-- standard_cat/
|   |   |-- standard_dog/
|   |   `-- standard_snake/
|   |-- Level_2_Dir/
|   |   |-- standard_cat/
|   |   |-- standard_moon/
|   |   `-- standard_sun/
|   `-- Level_3_Dir/
|       |-- standard_man/
|       |-- standard_moon/
|       `-- standard_woman/
`-- script

13 directories, 1 file
$ bash script
Top_Dir/Level_1_Dir/standard_cat/
Top_Dir/Level_2_Dir/standard_cat/
Top_Dir/Level_2_Dir/standard_moon/
Top_Dir/Level_3_Dir/standard_moon/

To support older bash releases, you may choose to store the unique basenames of the directories and the counts of how many times each basename has been seen in two separate ordinary arrays. This requires a linear search in each loop:

shopt -s nullglob dotglob

set -- Top_Dir/*/*/

names=()
counts=()
for dirpath do
        name=$( basename "$dirpath" )

        found=false
        for i in "${!names[@]}"; do
                if [[ ${names[i]} == "$name" ]]; then
                        found=true
                        break
                fi
        done

        if "$found"; then
                counts[i]=$(( counts[i] + 1 ))
        else
                names+=( "$name" )
                counts+=( 1 )
        fi
done

for dirpath do
        name=$( basename "$dirpath" )

        for i in "${!names[@]}"; do
                if [[ ${names[i]} == "$name" ]]; then
                        [[ ${counts[i]} -gt 1 ]] && printf '%s\n' "$dirpath"
                        break
                fi
        done
done
-1

This worked using bash on Ubuntu. It only matches duplicate directories irrespective of depth in the tree. The portion within the $() builds a list of pipe-separated directory names by counting duplicates in the last column of ls -l. This pipe-separated list is filtered using grep over the list of all directories. Also, not accounting for other files (didn't use whole word matches etc.)

> ls -lR Top_Dir/ | grep -E $(ls -lR Top_Dir/ | grep ^d | rev | cut -d" " -f1 | rev | sort | uniq -d | head -c -1 | tr '\n' '|') | grep -v ^d | sed 's/://'

Top_Dir/Level_1_Dir/standard_cat

Top_Dir/Level_2_Dir/standard_cat

Top_Dir/Level_2_Dir/standard_moon

Top_Dir/Level_3_Dir/standard_moon

4
  • First, thanks for your reply. When running the command I am getting a head: illegal byte count -- -1 error. Any thoughts on that? I have been paying with the head command but not seeing what is wrong with it.
    – dino
    Jun 10 '16 at 0:52
  • It's probably specific to your version of head. The link also contains other solutions to truncate the last line.
    – Jedi
    Jun 10 '16 at 0:59
  • 1
    did not function for special characters / spaces in folder names
    – Kevin
    Dec 15 '20 at 18:55
  • This explicitly disqualifies filenames with spaces, and also filenames that contain special characters, such as tabs, newlines, or any non-printable character (ls -l would show these as ?). Also note that the user is on macOS, which does not generally have GNU tools installed.
    – Kusalananda
    Mar 3 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.