Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
Looking for deduplication tools on Linux there are plenty, see e.g. this wiki page.
Allmost all scripts do either only detection, printing the duplicate file names or removing duplicate files by hardlinking them to a single copy.
With the rise of btrfs there would be another option: creating a CoW (copy-on-write) copy of a file (like cp reflink=always). I have not found any tool that does this, is anyone aware of tool that does this?
I wrote bedup for this purpose. It combines incremental btree scanning with CoW-deduplication. Best used with Linux 3.6, where you can run:
sudo bedup dedup
I tried bedup. While good (and has some useful differentiated features that may make it the best choice for many), it seems to scan the entirety of all target files for checksums.
Which is painfully slow.
Other programs on the other hand, such as rdfind and rmlint, scan differently.
rdfind has an "experimental" feature for using btrfs reflink. (And "solid" options for hardlinks, symlinks, etc.)
rmlint has "solid" options for btrfs clone, reflink, regular hardlinks, symlinks, delete, and your own custom commands.
But more importantly, rdfind and rmlint are significantly faster. As in, orders of magnitude. Rather than scanning all target files for checksums, it does this, approximately:
Other advantages of rmlint I'm aware of:
I compared rmlint with deduperemove--which also blindly scans all of every target file for checksums. Duperemove took several days on my volume to complete (4 I think), going full-tilt. fmlint took a few hours to identify duplicates, then less than a day to dedup them with Btrfs clone.
(That said, anyone making the effort to write and support quality, robust software and give it away for free, deserves major kudos!)
Btw: You should avoid deduping using regular hardlinks as a "general" dedup solution, at all costs.
While hardlinks can be extremely handy in certain targeted use cases (e.g. individual files or with a tool that can scan for specific file types exceeding some minimum size--or as part of many free and commercial backup/snapshot solutions), it can be disastrous for "deduplication" on a large general-use filesystem. The reason is that most users may have thousands of files on their filesystem, that are binary identical, but functionally completely different.
For example, many programs generate template and/or hidden settings files (sometimes in every single folder it can see), that are initially identical--and most remain so, until you, the user, need them not to be.
As a specific illustration: Photo thumbnail cache files, which countless programs generate in the folder containing the photos (and for good reason--portability), can take hours or days to generate but then make using a photo app a breeze. If those initial cache files are all hardlinked together, then you later open the app on a directory and it builds a large cache...then guess what: Now EVERY folder that has a previously hardlinked cache, now has the wrong cache. Potentially, with disastrous results that may result in accidental data destruction. And also potentially in a way that explodes a backup solution that isn't hardlink-aware.
Furthermore, it can ruin entire snapshots. The whole point of snapshots is so that the "live" version can continue to change, with the ability to roll back to a previous state. If everything is hardlinked together though...you "roll back" to the same thing.
The good news though is that deduping with Btrfs clone/reflink, can undo that damage (I think--since during the scan, it should see hardlinked files as identical...unless it has logic to not consider hardlinks. It probably depends on the specific utility doing the deduping.)
