I'm looking for a file system that stores files by block content, therefore similar files would only take one block. This is for backup purposes. It is, similar to what block-level backup storage proposes such as zbackup, but I'd like a Linux file system that allows to do that transparently.
Assuming your question is about data deduplication, there are a few file systems which support that on Linux:
- ZFS, with online deduplication (so data is deduplicated as it is stored), but with extreme memory requirements which make the feature hard to use in practice;
- Btrfs, with “only” out-of-band deduplication, albeit with tightly-integrated processes which provide reasonably quick deduplication after data is stored;
- SquashFS, but that probably doesn’t fit your requirements because it’s read-only.
XFS is supposed to get deduplication at some point, and Btrfs is also supposed to get online deduplication.
Keep an eye on Wikipedia’s file system comparison to see when this changes.
The S3QL filesystem has block level deduplication. It's promoted as an S3 cloud storage solution but it also works very nicely on local storage.
Here's an example from part of our backups/archive server
s3qlstat /path/to/some/archives Directory entries: 12430247 Inodes: 6343756 Data blocks: 1357349 Total data size: 12.4 TB After de-duplication: 3.84 TB (30.92% of total) After compression: 3.71 TB (29.84% of total, 96.52% of de-duplicated) Database size: 1.29 GiB (uncompressed) Cache size: 0 bytes, 0 entries Cache size (dirty): 0 bytes, 0 entries Queued object removals: 0
The underlying storage that this filesystem uses
df -h /var/s3ql/part-of-archive Filesystem Size Used Avail Use% Mounted on /dev/sde 6.0T 3.8T 2.0T 66% /var/s3ql/part-of-archive
This tells me that the underlying storage is using a little under 4TB, but that it's storing around 12TB of deduplicated data. (My archives have quite a lot of duplicated blocks in them. Unsurprisingly I hope.) The compression layer is disabled here; if I was using true S3 storage I would have left it enabled.
The SQLite database that manages the filesystem itself is just over 1GB, which is quite large, but as I'm using this for archiving rather than full-on high demand production use is fine.