Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

What are available solutions for block level or more detailed deduplication ?

There are file-based ones - with "Copy-On-Write" approach.

I'm looking for block level "copy-on-write", so I could periodically look for common blocks, or - preferably - parts of files, merge them and flag for CoW use manner. Is there sth like this available, or does it still need to be created ? I am not sure if Btrfs deduplication is block/file/subpart level ? There is LessFS, but I'am not sure what level of deduplication does it provide ? Maybe other solution?

share|improve this question
    
I don't understand why this question is being up-voted. Googling for "linux deduplication" turns up a list of filesystems and products which we don't need to duplicate here. –  Kyle Jones Mar 22 '12 at 18:01
    
Such googling returns a lot of solutions working on files level. What the purpose of this question is, an advice which of them fits into my purpose - preferably to make possible to de-duplicate any two parts of files, not necessary adjusted to block level. If such solution is not available, than block level. Another thing is that a lot of stuff makes impression of experimental - I count on experience of other users to advice some mature solution. –  Grzegorz Wierzowiecki Mar 22 '12 at 19:05
1  
@GrzegorzWierzowiecki not sure it fits the bill, but check out cyphertite, formally epitome: peereboom.us/epitome cyphertite.com –  gabe. Mar 22 '12 at 23:42

1 Answer 1

As block level deduplication goes, I think ZFS is the uncontested best implementation out currently. It really isn't designed for after-the-fact optimization, because its deduplication (if turned on) is built directly into the read/write functions. Because of this, it can be a bit memory expensive under load, in trying to keep the most relevant portions of the deduplication table in memory, but ZFS is good at restricting itself to consuming not much more than 50% of memory, which depending on quantity of memory installed, could seem quite arbitrary (50% of 2Gb vs 50% of 64Gb, especially if few-if-any user tasks needing memory).

Depending on what you're looking to use it in, you've got some options:

OpenIndiana appears to have some good Desktop and Server options, based on Solaris

FreeBSD (since 9.0) has a pretty advanced version of ZFS (which includes deduplication) built in to it. One notable FreeBSD (then MonoWall) derived distribution is NAS4Free, which makes making a NAS pretty easy.

Linux has a few options, some with dedup, others without. Since you're looking for dedup, the most notable I've seen is zfsonlinux. I'm not sure what their progress is, or how stable their project is, but it definitely looks promising.

As to anything with partial block deduplication, I have seen NOTHING so far that reports an ability to do that.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.