Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have some complex read-only data in my file system. It contains thousands of snapshots of certain revisions of a svn repository, and the output of regression tests. Identical files between snapshots are already de-duplicated using hard links. This way, the storage capacity doesn't need to be large, but it still consumes a lot of inodes, and this makes fsck painfully long for my main file system.

I'd like to move these data to another file system, so that it doesn't affect the main file system too much. Do you have suggestions? Squashfs seems to be a possible choice, but I'll have to check if it can handle hard links efficiently.

share|improve this question
1  
Which OS? Are you willing to set up a file server with a different OS? –  Kevin Cantu Sep 28 '10 at 16:31

4 Answers 4

up vote 4 down vote accepted

If it's abot fsck slowness, did you try ext4? They added a few features to it that make fsck really quick by not looking at unused inodes:

Fsck is a very slow operation, especially the first step: checking all the inodes in the file system. In Ext4, at the end of each group's inode table will be stored a list of unused inodes (with a checksum, for safety), so fsck will not check those inodes. The result is that total fsck time improves from 2 to 20 times, depending on the number of used inodes (http://kerneltrap.org/Linux/Improving_fsck_Speeds_in_Ext4). It must be noticed that it's fsck, and not Ext4, who will build the list of unused inodes. This means that you must run fsck to get the list of unused inodes built, and only the next fsck run will be faster (you need to pass a fsck in order to convert a Ext3 filesystem to Ext4 anyway). There's also a feature that takes part in this fsck speed up - "flexible block groups" - that also speeds up filesystem operations.

share|improve this answer
    
Looks promising. I'll give it a try. –  Wei-Yin Sep 29 '10 at 11:13
    
I see you use Ext3 now. You can convert ext3 into ext4 trivially (there are boatloads of howtos out there, it's basically just mounting the ext3 partition with a special parameter, then it's ext4 forever). –  tante Sep 29 '10 at 11:23

Btrfs has native support for snapshots, so you wouldn't have to use hard links for deduplication. You could recreate your current setup by creating a btrfs filesystem and loading it with the earliest revision that you need, and taking a snapshot, and then revving the repository forward to each point in time that you need a snapshot of and taking a snapshot at each step. This should be more efficient than hard links, and simpler to set up as well.

I also think (though I'm far from sure of this) that squashfs deduplicates files transparently, so even if it doesn't handle hard links, you'd still see benefits. If you never need to change the data in the filesystem, then squashfs is probably the way to go, since fsck could then be replaced by md5sum ;)

share|improve this answer

I would prefer XFS since I have very good experiences with this file system. But I really recommend, you make a test with your data and all filesystems suggested.

share|improve this answer
1  
Thanks for your suggestion. I'm using ext3 right now. Is fsck faster on XFS than ext3? –  Wei-Yin Sep 29 '10 at 11:12
1  
Yes, the fsck is faster. But as tante told as well, you should migrate it to ext4. –  ddeimeke Sep 30 '10 at 4:45

I know of several shops which use a DataDomain for exactly that purpose.

Your archival script can be very simple (tar or rsync and cron, for example), and you don't need to worry about managing hard links, or directories which cannot be hardlinked on most filesystems. No need for incremental copies except to conserve bandwidth. All the magic happens underneath within the block layer. It's not unusual to host 15-20TB worth of virtual data while only using 1-2TB worth of real disk space. You'll still have plenty left over for your disk backups.

The data would be served over NFS or iSCSI, but I'm not sure if that is a problem

When FreeBSD gets ZFS v23, deduplication will be available for the rest of us.

share|improve this answer
    
Thank you. Live deduplication is pretty nice. –  Wei-Yin Oct 5 '10 at 10:40
    
The use of deduplication is both memory expensive (with probability of bad side-effects if memory ends up running out, which happens more often then you might imagine), but also is only really helpful in certain (probably corporate) use cases. Use of ZFS snapshots would work though. –  killermist Aug 6 '12 at 18:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.