Finding duplicate files and replace them with symlinks

Question

I'm trying to find a way to check inside a given directory for duplicate files (even with different names) and replace them with symlinks pointing to the first occurrence. I've tried with fdupes but it just lists those duplicates.
That's the context: I'm customizing an icon theme to my liking, and I've found that many icons, even if they have different names and different locations inside their parent folder, and are used for different purposes, basically are just the same picture. Since applying the same modification twenty or thirty times is redundant when just one is really necessary, I want to keep just one image and symlink all the others.

As an example, if I run fdupes -r ./ inside the directory testdir, it might return to me the following results:

./file1.png
./file2.png
./subdir1/anotherfile.png
./subdir1/subdir2/yetanotherfile.png

Given this output, I'd like to keep just the file file1.png, delete all the others and replace them with symlinks pointing to it, while maintaining all original file names. So file2.png will retain its name, but will become a link to file1.png instead of being a duplicate.

Those links should not point to an absolute path, but should be relative to the parent testdir directory; i.e. yetanotherfile.png will be point to ../../file1.png, not to /home/testuser/.icons/testdir/file1.png

I'm interested both in solutions that involve a GUI and CLI. It is not mandatory to use fdupes I've cited it because it is a tool that I know, but I'm open to solutions that use other tools as well.

I'm pretty sure that a bash script to handle all of this should not be that difficult to create, but I'm not expert enough to find out how to write it myself.

Answer 1

If you don't fancy much scripting then I can recommend rdfind. Which will scan given directories for duplicate files and replace them with either hard or symbolic links. I've used it for deduplicating my Ruby gems directory with great success. It's available in Debian/Ubuntu.

Answer 2

I had a similar situation, but in my case the symbolic link should point to a relative path so I wrote this python script to do the trick:

#!/usr/bin/env python
# Reads fdupes(-r -1) output and create relative symbolic links for each duplicate
# usage: fdupes -r1 . | ./lndupes.py

import os
from os.path import dirname, relpath, basename, join
import sys

lines = sys.stdin.readlines()

for line in lines:
    files = line.strip().split(' ')
    first = files[0]
    print "First: %s "% first
    for dup in files[1:]:
        rel = os.path.relpath(dirname(first), dirname(dup))
        print "Linking duplicate: %s to %s" % (dup, join(rel,basename(first)))
        os.unlink(dup)
        os.symlink(join(rel,basename(first)), dup)

For each input line ( which is a list of files ) the script splits the file list (whitespace separated), gets the relative path from each file to the first one and then creates the symlink.

Answer 3

First; Is there a reason you need to use symlinks and not the usual hardlinks? I am having a hard time understanding the need for symlinks with relative paths. Here is how I would solve this problem:

I think the Debian (Ubuntu) version of fdupes can replace duplicates with hard links using the -L option, but I don't have a Debian installation to verify this.

If you do not have a version with the -L option you can use this tiny bash script I found on commandlinefu.
Note that this syntax will only work in bash.

fdupes -r -1 path | while read line; do master=""; for file in ${line[*]}; do if [ "x${master}" == "x" ]; then master=$file; else ln -f "${master}" "${file}"; fi; done; done

The above command will find all duplicate files in "path" and replace them with hardlinks. You can verify this by running ls -ilR and looking at the inode number. Here is a samle with ten identical files:

$ ls -ilR

total 20
3094308 -rw------- 1 username group  5 Sep 14 17:21 file
3094311 -rw------- 1 username group  5 Sep 14 17:21 file2
3094312 -rw------- 1 username group  5 Sep 14 17:21 file3
3094313 -rw------- 1 username group  5 Sep 14 17:21 file4
3094314 -rw------- 1 username group  5 Sep 14 17:21 file5
3094315 drwx------ 1 username group 48 Sep 14 17:22 subdirectory

./subdirectory:
total 20
3094316 -rw------- 1 username group 5 Sep 14 17:22 file
3094332 -rw------- 1 username group 5 Sep 14 17:22 file2
3094345 -rw------- 1 username group 5 Sep 14 17:22 file3
3094346 -rw------- 1 username group 5 Sep 14 17:22 file4
3094347 -rw------- 1 username group 5 Sep 14 17:22 file5

All the files have separate inode numbers, making them separate files. Now lets deduplicate them:

$ fdupes -r -1 . | while read line; do j="0"; for file in ${line[*]}; do if [ "$j" == "0" ]; then j="1"; else ln -f ${line// .*/} $file; fi; done; done
$ ls -ilR
.:
total 20
3094308 -rw------- 10 username group  5 Sep 14 17:21 file
3094308 -rw------- 10 username group  5 Sep 14 17:21 file2
3094308 -rw------- 10 username group  5 Sep 14 17:21 file3
3094308 -rw------- 10 username group  5 Sep 14 17:21 file4
3094308 -rw------- 10 username group  5 Sep 14 17:21 file5
3094315 drwx------  1 username group 48 Sep 14 17:24 subdirectory

./subdirectory:
total 20
3094308 -rw------- 10 username group 5 Sep 14 17:21 file
3094308 -rw------- 10 username group 5 Sep 14 17:21 file2
3094308 -rw------- 10 username group 5 Sep 14 17:21 file3
3094308 -rw------- 10 username group 5 Sep 14 17:21 file4
3094308 -rw------- 10 username group 5 Sep 14 17:21 file5

The files now all have the same inode number, meaning they all point to the same physical data on disk.

I hope this solves your problem or at least points you in the right direction!

Answer 4

So, the answer given by arnefm (that's been copied all over the internet) does not deal with spaces in file names. I've written a script that deals with spaces in files.

#!/bin/bash
fdupes -r -1 CHANGE_THIS_PATH | sed -e 's/\(\w\) /\1|/g' -e 's/|$//' > files
while read line; do
        IFS='|' read -a arr <<< "$line"
        orig=${arr[0]}
        for ((i = 1; i < ${#arr[@]}; i++)); do
                file="${arr[$i]}"
                ln -sf "$orig" "$file"
        done 
done < files

What this does is find dupes and write them PIPE separated into a file named 'files'.

Then it reads the file back, line by line, into an array, and each element of the array is delimited by the PIPE.

It then iterates over all the non-first elements of the array, replacing the file with a symlink to the first element.

The external file ('files') could be removed, if the fdupes command is executed in a subshell, that's read directly by the while, but this way seems clearer.

Answer 5

Some caveats up front:

• BASH specific
• No space in file names
• Assumes each line contains 2 files at most.

fdupes -1r common/base/dir | while read -r -a line ; do ln -sf $(realpath --relative-to ${line[1]} ${line[0]}) ${line[1]}; done

If more than 2 files are duplicates (e.g. file1 file2 file3) than we need to create a symlink for each pair - treat file1,file2 and file1,file3 as 2 separate cases:

if [[ ${#line[@]} -gt 2 ]] ;then 
  ln -sf $(realpath --relative-to ${line[1]} ${line[0]}) ${line[1]} 
  ln -sf $(realpath --relative-to ${line[2]} ${line[0]}) ${line[2]} 
  ...
fi

Expending this to automatically handle an arbitrary number of duplicates per line will take a little more effort.

Another approach would be to first create symlinks to absolute paths, then convert them:

fdupes -1r /absolute/path/common/base/dir | while read -r -a line ; do ln -sf ${line[0]} ${line[1]}; done
chroot /absolute/path/common/base/dir ; symlinks -cr .

This is based on the answer by @Gilles: https://unix.stackexchange.com/a/100955/77319