filesystem for archiving

Question

I have some complex read-only data in my file system. It contains thousands of snapshots of certain revisions of a svn repository, and the output of regression tests. Identical files between snapshots are already de-duplicated using hard links. This way, the storage capacity doesn't need to be large, but it still consumes a lot of inodes, and this makes fsck painfully long for my main file system.

I'd like to move these data to another file system, so that it doesn't affect the main file system too much. Do you have suggestions? Squashfs seems to be a possible choice, but I'll have to check if it can handle hard links efficiently.

Answer 1

Btrfs has native support for snapshots, so you wouldn't have to use hard links for deduplication. You could recreate your current setup by creating a btrfs filesystem and loading it with the earliest revision that you need, and taking a snapshot, and then revving the repository forward to each point in time that you need a snapshot of and taking a snapshot at each step. This should be more efficient than hard links, and simpler to set up as well.

I also think (though I'm far from sure of this) that squashfs deduplicates files transparently, so even if it doesn't handle hard links, you'd still see benefits. If you never need to change the data in the filesystem, then squashfs is probably the way to go, since fsck could then be replaced by md5sum ;)

Answer 2

I would prefer XFS since I have very good experiences with this file system. But I really recommend, you make a test with your data and all filesystems suggested.

Answer 3

If it's abot fsck slowness, did you try ext4? They added a few features to it that make fsck really quick by not looking at unused inodes:

Fsck is a very slow operation, especially the first step: checking all the inodes in the file system. In Ext4, at the end of each group's inode table will be stored a list of unused inodes (with a checksum, for safety), so fsck will not check those inodes. The result is that total fsck time improves from 2 to 20 times, depending on the number of used inodes (http://kerneltrap.org/Linux/Improving_fsck_Speeds_in_Ext4). It must be noticed that it's fsck, and not Ext4, who will build the list of unused inodes. This means that you must run fsck to get the list of unused inodes built, and only the next fsck run will be faster (you need to pass a fsck in order to convert a Ext3 filesystem to Ext4 anyway). There's also a feature that takes part in this fsck speed up - "flexible block groups" - that also speeds up filesystem operations.

Answer 4

I know of several shops which use a DataDomain for exactly that purpose.

Your archival script can be very simple (tar or rsync and cron, for example), and you don't need to worry about managing hard links, or directories which cannot be hardlinked on most filesystems. No need for incremental copies except to conserve bandwidth. All the magic happens underneath within the block layer. It's not unusual to host 15-20TB worth of virtual data while only using 1-2TB worth of real disk space. You'll still have plenty left over for your disk backups.

The data would be served over NFS or iSCSI, but I'm not sure if that is a problem

When FreeBSD gets ZFS v23, deduplication will be available for the rest of us.