Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm looking for an easy way (a command or series of commands, probably involving find) to find duplicate files in two directories, and replace the files in one directory with hardlinks of the files in the other directory.

Here's the situation: This is a file server which multiple people store audio files on, each user having their own folder. Sometimes multiple people have copies of the exact same audio files. Right now, these are duplicates. I'd like to make it so they're hardlinks, to save hard drive space.

share|improve this question
11  
One problem you may run into with hardlinks is if somebody decides to do something to one of their music files that you've hard-linked they could inadvertently be affecting other people's access to their music. –  Steven D Oct 13 '10 at 2:48
3  
another problem is that two different files containing "Some Really Great Tune", even if taken from the same source with the same encoder will very likely not be bit-for-bit identical. –  msw Oct 13 '10 at 2:57
1  
better sollution might be to have a public music folder... –  NixNinja Oct 13 '10 at 7:08
    
Use symlinks then: Copy the MP3 Files to a central repo and symlink them into the user's folder, "deleting" a file would just mean deleting the symlink. (But Stefan's idea of the public music folder is better) –  tante Oct 13 '10 at 7:52
1  
show 1 more comment

14 Answers

up vote 20 down vote accepted

There is a perl script at http://cpansearch.perl.org/src/ANDK/Perl-Repository-APC-2.002/eg/trimtrees.pl which does exactly what you want:

Traverse all directories named on the command line, compute MD5 checksums and find files with identical MD5. IF they are equal, do a real comparison if they are really equal, replace the second of two files with a hard link to the first one.

share|improve this answer
    
Sounds perfect, thanks!! I'll try it and accept if it works as described! –  Josh Oct 12 '10 at 20:09
2  
This did exactly what I asked for. However I believe that ZFS with dedup will eventually be the way to do, since I did find that the files had slight differences so only a few could be hardlinked. –  Josh Dec 8 '10 at 20:13
add comment

Use the fdupes tool:

fdupes -r /path/to/folder gives you a list of duplicates in the directory (-r makes it recursive). The output looks like this:


filename1
filename2

filename3
filename4
filename5


with filename1 and filename2 being identical and filename3, filename4 and filename5 also being identical.

share|improve this answer
8  
In Debian, you can use the -L option to replace duplicate files with hardlinks (source) –  Michaël Witrant Jul 21 '11 at 15:16
    
Ubuntu Note: As of September 2013, it hasn't had a stable release (it is on 1.50-PR2-3), so the update doesn't appear in ubuntu yet. –  Stuart Axon Aug 28 '13 at 14:19
2  
I just tried installing fdupes_1.50-PR2-4 on both Ubuntu and Debian, neither has the -L flag. Luckily building from github.com/tobiasschulz/fdupes was super easy. –  neu242 Aug 30 '13 at 15:07
add comment

Since your main target is to save disk space, there is another solution: de-duplication (and probably compression) on file system level. Compared with the hard-link solution, it does not have the problem of inadvertently affecting other linked files.

ZFS has dedup (block-level, not file-level) since pool version 23 and compression since long time ago. If you are using linux, you may try zfs-fuse, or if you use BSD, it is natively supported.

share|improve this answer
    
This is probably the way I'll go eventually, however, does BSD's ZFS implementation do dedup? I thought it did not. –  Josh Dec 8 '10 at 20:14
    
In addition, the HAMMER filesystem on DragonFlyBSD has deduplication support. –  hhaamu Jul 15 '12 at 17:48
    
I used zfs-fuse for a backup partition because of its deduplication and it took days, literally days to copy less than 1TB of data onto it. I can't begin to imagine how zfs-fuse would perform as a home partition or something. –  Christian Jun 14 '13 at 15:04
add comment

This is one of the functions provided by "fslint" -- http://en.flossmanuals.net/FSlint/Introduction

share|improve this answer
2  
The -m will hardlink duplicates together, -d will delete all but one, and -t will dry run, printing what it would do –  Azendale Oct 29 '12 at 5:57
    
On Ubuntu here is what to do: sudo apt-get install fslint /usr/share/fslint/fslint/findup -m /your/directory/tree (directory /usr/share/fslint/fslint/ is not in $PATH by default) –  Jocelyn Sep 8 '13 at 15:38
add comment

I use hardlink from http://jak-linux.org/projects/hardlink/

share|improve this answer
1  
Nice hint, I am using on a regular base code.google.com/p/hardlinkpy but this was not updated for a while... –  meduz Apr 11 '12 at 19:09
    
This appears to be similar to the original hardlink on Fedora/RHEL/etc. –  Jack Douglas Jun 21 '12 at 8:43
add comment

To find duplicate files you can use duff.

Duff is a Unix command-line utility for quickly finding duplicates in a given set of files.

Simply run:

duff -r target-folder

To create hardlinks to those files automaticly, you will need to parse the output of duff with bash or some other scripting language.

share|improve this answer
    
Helpful, thanks! –  Josh Oct 12 '10 at 20:08
add comment

On modern Linux these days there's https://github.com/g2p/bedup which de-duplicates on a btrfs filesystem, but 1) without as much of the scan overhead, 2) files can diverge easily again afterwards.

share|improve this answer
add comment

I made a Perl script that does something similar to what you're talking about:

http://pastebin.com/U7mFHZU7

Basically, it just traverses a directory, calculating the SHA1sum of the files in it, hashing it and linking matches together. It's come in handy on many, many occasions.

share|improve this answer
2  
I hope to get around to trying this soon... why not upload it on CPAN... App::relink or something –  xenoterracide Feb 7 '11 at 11:12
add comment

I've used many of the hardlinking tools for Linux mentioned here. I too am stuck with ext4 fs, on Ubuntu, and have been using its cp -l and -s for hard/softlinking. But lately noticed the lightweight copy in the cp man page, which would imply to spare the redundant disk space until one side gets modified:

   --reflink[=WHEN]
          control clone/CoW copies. See below

       When  --reflink[=always]  is specified, perform a lightweight copy, where the 
data blocks are copied only when modified.  If this is not possible the
       copy fails, or if --reflink=auto is specified, fall back to a standard copy.
share|improve this answer
    
I think I will update my cp alias to always include the --reflink=auto parameter now –  Marcos Mar 14 '12 at 14:08
1  
Does ext4 really support --reflink? –  Jack Douglas Jun 21 '12 at 8:42
4  
This is supported on btrfs and OCFS2. It is only possible on copy-on-write filesystems, which ext4 is not. btrfs is really shaping up. I love using it because of reflink and snapshots, makes you less scared to do mass operations on big trees of files. –  clacke Jul 3 '12 at 18:57
add comment

If you want to replace duplicates by Hard Links on mac or any UNIX based system, you can try SmartDupe http://sourceforge.net/projects/smartdupe/ am developing it

share|improve this answer
2  
Can you expand on how “smart” it is? –  Stéphane Gimenez Nov 4 '12 at 13:25
add comment

rdfind does exactly what you ask for (and in the order johny why lists). Makes it possible to delete duplicates, replace them with either soft or hard links. Combined with symlinks you can also make the symlink either absolute or relative. You can even pick checksum algorithm (md5 or sha1).

Since it is compiled it is faster than most scripted solutions: time on a 15 GiB folder with 2600 files on my Mac Mini from 2009 returns this

9.99s user 3.61s system 66% cpu 20.543 total

(using md5).

Available in most package handlers (e.g. MacPorts for Mac OS X).

share|improve this answer
    
+1 I used rdfind and loved it. It has a -dryrun true option that will let you know what it would have done. Replacing duplicates with hard links is as simple as -makehardlinks true. It produced a nice log and it let me know how much space was freed up. Plus, according to the author's benchmark, rdfind is faster than duff and fslint. –  Daniel Trebbien Dec 29 '13 at 20:49
add comment

Since I'm not a fan of Perl, here's a bash version:

#!/bin/bash

DIR="/path/to/big/files"

find $DIR -type f -exec md5sum {} \; | sort > /tmp/sums-sorted.txt

OLDSUM=""
IFS=$'\n'
for i in `cat /tmp/sums-sorted.txt`; do
 NEWSUM=`echo "$i" | sed 's/ .*//'`
 NEWFILE=`echo "$i" | sed 's/^[^ ]* *//'`
 if [ "$OLDSUM" == "$NEWSUM" ]; then
  echo ln -f "$OLDFILE" "$NEWFILE"
 else
  OLDSUM="$NEWSUM"
  OLDFILE="$NEWFILE"
 fi
done

This finds all files with the same checksum (whether they're big, small, or already hardlinks), and hardlinks them together.

This can be greatly optimized for repeated runs with additional find flags (eg. size) and a file cache (so you don't have to redo the checksums each time). If anyone's interested in the smarter, longer version, I can post it.

NOTE: As has been mentioned before, hardlinks work as long as the files never need modification, or to be moved across filesystems.

share|improve this answer
    
How can I change your script, so that instead of hardlinking it, it will just delete the duplicate files and will add an entry to a CSV file the deleted file -> Lined File. . ??? –  MR.GEWA Jan 12 '13 at 12:17
    
Sure. The hard link line: echo ln -f "$OLDFILE" "$NEWFILE" Just replaces the duplicate file with a hard link, so you could change it rm the $NEWFILE instead. –  seren Jan 13 '13 at 4:15
    
and how on next line, write in some text file somehow $OLDFILE-> NEWFILE ??? –  MR.GEWA Jan 13 '13 at 13:12
    
Ahh, right. Yes, add a line after the rm such as: echo "$NEWFILE" >> /var/log/deleted_duplicate_files.log –  seren Jan 14 '13 at 19:28
    
thanks @seren so much! –  MR.GEWA Jan 15 '13 at 15:03
add comment

Seems to me that checking the filename first could speed things up. If two files lack the same filename then in many cases I would not consider them to be duplicates. Seems that the quickest method would be to compare, in order:

  • filename
  • size
  • md5 checksum
  • byte contents

Do any methods do this? Look at duff, fdupes, rmlint, fslint, etc.

The following method was top-voted on commandlinefu.com: Find Duplicate Files (based on size first, then MD5 hash)

Can filename comparison be added as a first step, size as a second step?

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | \
  xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | \
  sort | uniq -w32 --all-repeated=separate
share|improve this answer
add comment
aptitude show hardlink

Description: Hardlinks multiple copies of the same file Hardlink is a tool which detects multiple copies of the same file and replaces them with hardlinks.

The idea has been taken from http://code.google.com/p/hardlinkpy/, but the code has been written from scratch and licensed under the MIT license. Homepage: http://jak-linux.org/projects/hardlink/

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.