What are available solutions for block level or more detailed deduplication ?

There are file-based ones - with "Copy-On-Write" approach.

I'm looking for block level "copy-on-write", so I could periodically look for common blocks, or - preferably - parts of files, merge them and flag for CoW use manner. Is there something like this available, or does it still need to be created ? I am not sure if Btrfs deduplication is block/file/subpart level ? There is LessFS, but I'am not sure what level of deduplication does it provide ? Maybe other solution?

  • I don't understand why this question is being up-voted. Googling for "linux deduplication" turns up a list of filesystems and products which we don't need to duplicate here. – Kyle Jones Mar 22 '12 at 18:01
  • Such googling returns a lot of solutions working on files level. What the purpose of this question is, an advice which of them fits into my purpose - preferably to make possible to de-duplicate any two parts of files, not necessary adjusted to block level. If such solution is not available, than block level. Another thing is that a lot of stuff makes impression of experimental - I count on experience of other users to advice some mature solution. – Grzegorz Wierzowiecki Mar 22 '12 at 19:05
  • 1
    @GrzegorzWierzowiecki not sure it fits the bill, but check out cyphertite, formally epitome: peereboom.us/epitome cyphertite.com – gabe. Mar 22 '12 at 23:42

As block level deduplication goes, I think ZFS is the uncontested best implementation out currently. It really isn't designed for after-the-fact optimization, because its deduplication (if turned on) is built directly into the read/write functions. Because of this, it can be a bit memory expensive under load, in trying to keep the most relevant portions of the deduplication table in memory, but ZFS is good at restricting itself to consuming not much more than 50% of memory, which depending on quantity of memory installed, could seem quite arbitrary (50% of 2Gb vs 50% of 64Gb, especially if few-if-any user tasks needing memory).

Depending on what you're looking to use it in, you've got some options:

OpenIndiana appears to have some good Desktop and Server options, based on Solaris

FreeBSD (since 9.0) has a pretty advanced version of ZFS (which includes deduplication) built in to it. One notable FreeBSD (then MonoWall) derived distribution is NAS4Free, which makes making a NAS pretty easy.

Linux has a few options, some with dedup, others without. Since you're looking for dedup, the most notable I've seen is zfsonlinux. I'm not sure what their progress is, or how stable their project is, but it definitely looks promising.

As to anything with partial block deduplication, I have seen NOTHING so far that reports an ability to do that.


Your question is a tiny bit confusing due to the term "blocks", which is a very overloaded word when it comes to disks and filesystems. (But your surrounding context helps clarify.) Btrfs doesn't deal with fixed-size filesystem "blocks", it deals with variable-sized "extents". (Though, confusingly, also defines variable-size block zones.) ZFS deals with filesystem "blocks", in part or primarily because doing so presents significantly easier problems to solve. Both Btrfs and ZFS are aware of disk-level "blocks", which are themselves abstractions. (Then we also have "block-level storage", which can be a semantically different meaning.) I might have those descriptions a little off, not clear enough, or not 100% accurate. (If you need clarity and 100% accuracy on the topic of blocks, pretend you didn't read that. If you just need a rough understanding to continue, then you should be good to go.) The main point of this answer is not to perfectly define "blocks", but the discussion below, which is much more in my wheelhouse.

As @killermist wrote, ZFS natively supports [ZFS] block-level deduplication.

It's not enabled by default in ZFS. Turning it on without enough memory involves a hefty performance hit. Also, anecdotally, ZFS needs a fair amount more than the "1gb RAM per 1tb storage" recommended rule-of-thumb, to fit the entire hashtable in RAM. But even so, depending on hardware you can still get upwards of 40 MB/s write speeds. I get that on 2008-era tech running ~2015-era drives. Perfectly acceptable to me for mostly archival data. The biggest drawback to ZFS deduplication, is that there isn't yet an elegant way to do it in "batch/offline" (or more accurately "out-of-band") mode, other than turning dedup on, copying everything to a new temp directory on the same filesystem, deleting the originals, then moving the (now-deduplicated) temp contents back. (Except that deleting the old files could take longer than the initial copy/dedup operation.) What I usually do is wait until I have to periodically rearchitect the array anyway in order to change the fundemantal layout, and copy from the old array to the new, with dedup on.

Btrfs deduplication is arguably a little sketchier, with only third-party utilities currently available to do the work. (But which use either well-supported kernel APIs, and/or a well supported option to cp; and either way requiring their own logic to determine duplicates, which one hopes is dead-on accurate.) One potential benefit though, is those utilities are "out-of-band". The cost to most of the utilities though, is that they kill performance while hammering away--which can take hours, days, even weeks to complete. (Personally I'd rather deal with always slower write performance of in-band ZFS deduplication, than hammering my HDDs for days on, say, end once per year.)

Two Btrfs solutions I'm aware of that deal with "blocks" (but in yet another definition) rather than files, are bees, and dduper.

Bees, for example, arbitrarily defines a "block" size for itself on first run, based on available memory and possibly other factors. (Though I'm probably misrepresenting its purpose, features, mechanism, and pros/cons, since I don't use it, I only evaluated it recently as an option.)

Bees is arguably slightly hybrid-ish, as it is designed to run continuously, and not hammer the disks quite so hard--though is still not technically "in-band" like ZFS dedup. It simply picks up duplicates after-the-fact and tries to deduplicate them with a light touch. Working with an arbitrarily-defined block size mean that, by design, it will fit the hashtable in RAM. The drawback (presumably) is that there may be extents in a "block" that are the same, but Bees may not dedup because the "blocks" they are in are otherwise different.

Keep in mind that even utilities that specifically perform "file"-level Btrfs deduplication (like bedup, duperemove, rmlint, and others), may still satisfy your requirements. I can't be sure, but it seems like they would. That's because even a "cp --reflink=always" command isn't really deduplicating "files". It is deduplicating Btrfs extents. When a reflinked "file" changes, Btrfs only un-deduplicates the extents that change, to their own unique extents. The rest of the file remains deduplicated. That's how large deduplicated files can still diverge as if their own unique files, but still be mostly deduplicated.

(This is also why it's so difficult to determine whether a "file" is reflinked or not, because that concept doesn't even really make sense. All of a file's extents may themselves be reflinked to other same-extents, a concept that does make sense, but that's coincidentally a particularly hard question to answer. Which is why, unless a Btrfs deduplication utility keeps track of what it has already deduplicated, it's not worth the effort to try to "detect" if a file has already been deduplicated. There's no attribute like refcount to inspect. It's easier to just deduplicate it again anyway. In contrast, determining whether an entire file is hardlinked the old-fashioned way, is trivial. Just check the st_nlink count for a given inode.)

The lack of "whole file clone" is in fact an intrinsic feature of all CoW filesystems that support "free" snapshots and/or deduplication, and is true whether dealing with Btrfs extents, ZFS blocks, or something else. Which is why either one can probably be an answer to your question. (There are at least three other CoW filesystems that can or are planned to be able to do all that, that I'm aware of: nilfs2, bcachefs, and xfs.)

Although you didn't mention this, no deduplication tech to my knowledge, is filetype-aware. In other words, no deduplicator knows to skip over *.jpg metadata and only consider the compressed image data for deduplication. Likewise, none of them consider file magic numbers (at least for determining what to consider for deduplication). That could be a killer feature--though certainly requiring constant, ongoing definition updates. And could be really hard to design, while also treating files as an abstract M:M collection of extents, blocks, etc.

  • To expand on this old answer, rmlint is now the uncontested btrfs deduper, at least currently. 1) It does smart compare, to avoid unnecessarily hashing duplicate candidates unless it has exausted other options; 2) The develop [and I believe master] branch supports incremental hashing, and 3) incremental deduping. Those last two are huge features. No other deduper provides all three features, which can literally cut days off subsequent runs. – Jim Jul 27 at 21:42
  • rmlint only considers identical files as candidates for deduplication, ignoring files with only partial duplicate ranges. – Tom Hale Aug 12 at 12:10
  • @tom-hale I don't understand your point? – Jim Aug 18 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.