I have some complex read-only data in my file system. It contains thousands of snapshots of certain revisions of a svn repository, and the output of regression tests. Identical files between snapshots are already de-duplicated using hard links. This way, the storage capacity doesn't need to be large, but it still consumes a lot of inodes, and this makes fsck painfully long for my main file system.

I'd like to move these data to another file system, so that it doesn't affect the main file system too much. Do you have suggestions? Squashfs seems to be a possible choice, but I'll have to check if it can handle hard links efficiently.

  • 1
    Which OS? Are you willing to set up a file server with a different OS? Sep 28, 2010 at 16:31

4 Answers 4


Btrfs has native support for snapshots, so you wouldn't have to use hard links for deduplication. You could recreate your current setup by creating a btrfs filesystem and loading it with the earliest revision that you need, and taking a snapshot, and then revving the repository forward to each point in time that you need a snapshot of and taking a snapshot at each step. This should be more efficient than hard links, and simpler to set up as well.

I also think (though I'm far from sure of this) that squashfs deduplicates files transparently, so even if it doesn't handle hard links, you'd still see benefits. If you never need to change the data in the filesystem, then squashfs is probably the way to go, since fsck could then be replaced by md5sum ;)


I would prefer XFS since I have very good experiences with this file system. But I really recommend, you make a test with your data and all filesystems suggested.

  • 1
    Thanks for your suggestion. I'm using ext3 right now. Is fsck faster on XFS than ext3?
    – Wei-Yin
    Sep 29, 2010 at 11:12
  • 1
    Yes, the fsck is faster. But as tante told as well, you should migrate it to ext4.
    – ddeimeke
    Sep 30, 2010 at 4:45

If it's abot fsck slowness, did you try ext4? They added a few features to it that make fsck really quick by not looking at unused inodes:

Fsck is a very slow operation, especially the first step: checking all the inodes in the file system. In Ext4, at the end of each group's inode table will be stored a list of unused inodes (with a checksum, for safety), so fsck will not check those inodes. The result is that total fsck time improves from 2 to 20 times, depending on the number of used inodes (http://kerneltrap.org/Linux/Improving_fsck_Speeds_in_Ext4). It must be noticed that it's fsck, and not Ext4, who will build the list of unused inodes. This means that you must run fsck to get the list of unused inodes built, and only the next fsck run will be faster (you need to pass a fsck in order to convert a Ext3 filesystem to Ext4 anyway). There's also a feature that takes part in this fsck speed up - "flexible block groups" - that also speeds up filesystem operations.

  • Looks promising. I'll give it a try.
    – Wei-Yin
    Sep 29, 2010 at 11:13
  • I see you use Ext3 now. You can convert ext3 into ext4 trivially (there are boatloads of howtos out there, it's basically just mounting the ext3 partition with a special parameter, then it's ext4 forever).
    – tante
    Sep 29, 2010 at 11:23

I know of several shops which use a DataDomain for exactly that purpose.

Your archival script can be very simple (tar or rsync and cron, for example), and you don't need to worry about managing hard links, or directories which cannot be hardlinked on most filesystems. No need for incremental copies except to conserve bandwidth. All the magic happens underneath within the block layer. It's not unusual to host 15-20TB worth of virtual data while only using 1-2TB worth of real disk space. You'll still have plenty left over for your disk backups.

The data would be served over NFS or iSCSI, but I'm not sure if that is a problem

When FreeBSD gets ZFS v23, deduplication will be available for the rest of us.

  • The use of deduplication is both memory expensive (with probability of bad side-effects if memory ends up running out, which happens more often then you might imagine), but also is only really helpful in certain (probably corporate) use cases. Use of ZFS snapshots would work though.
    – killermist
    Aug 6, 2012 at 18:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .