5

I'm looking for a file system that stores files by block content, therefore similar files would only take one block. This is for backup purposes. It is, similar to what block-level backup storage proposes such as zbackup, but I'd like a Linux file system that allows to do that transparently.

  • 4
    The feature I beleive you're looking for is called block-level "deduplication". ZFS is one FS that supports it. Beware it's costly in performance and memory and may not be worth it if you don't have a lot of duplication in the first place. With its snapshotting capability, zfs is also well suited for implementing backup strategies – Stéphane Chazelas Oct 1 '19 at 15:17
  • 1
    zbackup.org lists a number of "similar projects". – Stéphane Chazelas Oct 1 '19 at 15:19
  • github.com/dm-vdo is a(n out of tree) device mapper (not a filesystem, directly a block device on top of which which a filesystem or an LVM can be put) that does compression and deduplication. It's provided by Redhat, with a long term goal of upstream kernel integration. – A.B Oct 1 '19 at 20:54
  • ZFS de-duplication is an interesting feature, but (because it uses ARC to store the block checksums) in practice it is a good way to use lots of expensive RAM to minimise use of cheap disk. It will always be cheaper and more effective to add more disks to your zpool than to add more RAM. – cas Oct 2 '19 at 2:23
  • Using ZFS is good for lots of other reasons (snapshots, transparent compression, ecc, and much more) but de-duplication is IMO more of a gimmick than something that is actually useful. – cas Oct 2 '19 at 2:24
5

Assuming your question is about data deduplication, there are a few file systems which support that on Linux:

  • ZFS, with online deduplication (so data is deduplicated as it is stored), but with extreme memory requirements which make the feature hard to use in practice;
  • Btrfs, with “only” out-of-band deduplication, albeit with tightly-integrated processes which provide reasonably quick deduplication after data is stored;
  • SquashFS, but that probably doesn’t fit your requirements because it’s read-only.

XFS is supposed to get deduplication at some point, and Btrfs is also supposed to get online deduplication.

Keep an eye on Wikipedia’s file system comparison to see when this changes.

  • Yes, I know now that what I wanted is called "in-band deduplication" so the command "zfs set dedup=on mypool/myfs" enables it for zfs. However this runs on a small system, so I'm looking at less memory hungry alternatives in the Wikipedia list of FS comparisons with deduplication. Thanks! – MappaM Oct 1 '19 at 16:13
2

The S3QL filesystem has block level deduplication. It's promoted as an S3 cloud storage solution but it also works very nicely on local storage.

Here's an example from part of our backups/archive server

s3qlstat /path/to/some/archives
Directory entries:    12430247
Inodes:               6343756
Data blocks:          1357349
Total data size:      12.4 TB
After de-duplication: 3.84 TB (30.92% of total)
After compression:    3.71 TB (29.84% of total, 96.52% of de-duplicated)
Database size:        1.29 GiB (uncompressed)
Cache size:           0 bytes, 0 entries
Cache size (dirty):   0 bytes, 0 entries
Queued object removals: 0

The underlying storage that this filesystem uses

df -h /var/s3ql/part-of-archive
Filesystem      Size  Used Avail Use% Mounted on
/dev/sde        6.0T  3.8T  2.0T  66% /var/s3ql/part-of-archive

This tells me that the underlying storage is using a little under 4TB, but that it's storing around 12TB of deduplicated data. (My archives have quite a lot of duplicated blocks in them. Unsurprisingly I hope.) The compression layer is disabled here; if I was using true S3 storage I would have left it enabled.

The SQLite database that manages the filesystem itself is just over 1GB, which is quite large, but as I'm using this for archiving rather than full-on high demand production use is fine.

  • Do you think it is better in term of memory usage than the proposed ZFS solution? I guess at least the SQL database can be wrote back to disk if memory becomes short. – MappaM Oct 2 '19 at 19:50
  • ZFS dedupe is very memory greedy; if that's your solution then do the maths first to check you've got enough "spare" RAM for your proposed disk size. S3QL runs in user space so it's going to be slower than a native filesystem but it's not going to eat up all your memory. – roaima Oct 2 '19 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.