Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

At the moment I am using ArchiveMount to mount a 123,000 kb archive that contains more then 3 million files inside. So far it has been mounting for 5+ hours and still isn't finished.

Is there a better way to mount a .tar.gz file? I am trying to mount to a folder, and uncompressed it takes a few gigs. I don't even need write mode, just read-only is sufficient.

share|improve this question
    
There's also AVFS; I have no idea if it'll perform better. –  Gilles Nov 6 '11 at 16:25
5  
If your files were compressed as a squashfs module instead of as a tarball, then read-only access would be very quick - you just (loop) mount the squashfs module. Requires the squashfs-tools package. –  dru8274 Nov 7 '11 at 6:05

4 Answers 4

You could also create a compressed squashfs image

mksquashfs /etc squashfs.img -comp xz
mkdir img
mount -o squashfs,ro squashfs.img img

In order to do this you'll need to extract your tar.gz archvie.

The advantage is also that the image has better fault tolerancy than gz.

share|improve this answer

The problem here is with the format, the TAR (Tape ARchive) format is designed for sequential access, not random access. And gzip is a good complement to tar, since it is a stream based compression format, also not for random access.

So a high level tool that does not interact with the compressed blocks directly, will have to parse through the entire file every time it needs to read anything, first to get you the list of files, then perhaps the cache invalidates and it reads it again, and then for each file you copy off it might read through it again. You can make a tool that remembers the position of each file, and what blocks it needs to decompress to get it, but it seems that few have bothered with that.

If you want this to go faster, do a tar tzf file.tar.gz > filelist, open that file list in vim, gedit or whatever, remove the lines of files you do not need, save, and then extract them with tar xzf file.tar.gz -T filelist -C extracted/.

To get random access to a compressed file, you should use perhaps zip with posix extensions, rar, or as dru8274 suggested, squashfs, or even ZFS with compression turned on, or btrfs if btrfs has gotten compression to work at the time of reading.

share|improve this answer
    
To get random access to a compressed file, you could also use pixz. –  kubanczyk Jun 29 at 20:42

This will not cover all use-cases as it restricts use to a text-editor. But, if you only care about read-access, you might find this helpful for some situations. vim, when run on a tarball will show you the content hierarchy of the archive (similar to how it will display a file hierarchy if run on a directory). By selecting one of the files in the list, it will open the selected file in a read-only buffer.

Again, this doesn't necessarily offer access to images or other media, but if all you need is to see the contents or access text-based files only, then this should be helpful.

Note: this will not work on all archive formats.

share|improve this answer

My approach. If you have enough free disk space on an external USB drive or external/secondary HDD drive with enough space, then consider just extracting your .tar.gz file. Thinking you probably don't want 3 million files on your main system disk, as that could slow things down. I'd recommend that the external disk in this case have a filesystem that handles a huge number of files easily: thinking ReiserFS, ext4 (with dir_index option), XFS, maybe BtrFS. It could might take 1-2 hours to do the extract, but you could just go get lunch in the meantime or let it run overnight; when you come back, accessing the extracted files should be performant.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.