Faster alternative to ArchiveMount?

Question

At the moment I am using ArchiveMount to mount a 123,000 kb archive that contains more then 3 million files inside. So far it has been mounting for 5+ hours and still isn't finished.

Is there a better way to mount a .tar.gz file? I am trying to mount to a folder, and uncompressed it takes a few gigs. I don't even need write mode, just read-only is sufficient.

Answer 1

I wrote a faster alternative ratarmount, which "works for me", because this problem kept bugging me.

You can install and use it like this:

pip3 install --user ratarmount
ratarmount my-huge-tar.tar mount-folder
ls -la mount-folder # will show the contents of the tar top-level

When you are done you can unmount it like any FUSE mount:

fusermount -u mount-folder

Why is it faster than archivemount?

It depends on what you measure.

Here is a benchmark of memory footprint and required time for first mounting, as well as access times for a simple cat <file-in-tar> command and a simple find command.

Folders containing each 1k files were created and the number of folders is varied.

The lower left plot shows error bars indicating the minimum and maximum measured times for cat <file> for 10 randomly chosen files.

Pros

• Not shown in the benchmarks, but ratarmount can mount files with preexisting index sidecar files in under a second making it vastly more efficient compared to archivemount for every subsequent mounts.
• Ratarmount comes with a progress indicator, so that in contrast to archivemount, users do not have to try waiting for hours without getting any feedback.
• Getting file contents of a mounted archive is generally vastly faster than archivemount and fuse-archive and does not increase with the archive size or file count resulting in the largest observed speedups to be around 5 orders of magnitude!
• Mounting bzip2 and xz archives has actually become faster than archivemount and fuse-archive with ratarmount -P 0 on most modern processors because it actually uses more than one core for decoding those compressions. indexed_bzip2 supports block parallel decoding since version 1.2.0.
• Memory consumption of ratarmount is mostly less than archivemount and mostly does not grow with the archive size.
  • The gzip backend grows linearly with the archive size because the data for seeking is thousands of times larger than the simple two 64-bit offsets required for bzip2. If this becomes an issue, you can increase the seekpoint spacing using --gzip-seek-point-spacing <spacing in MiB> to reduce the total amount of points and therefore data to be generated.
  • The memory usage of the zstd backend only seems humungous because it uses mmap to open. The memory used by mmap is not even counted as used memory when showing the memory usage with free or htop.
• For empty files, mounting with ratarmount and archivemount does not seem be bounded by decompression nor I/O bandwidths but instead by the algorithm for creating the internal file index. This algorithm scales linearly for ratarmount and fuse-archive but seems to scale worse than even quadratically for archives containing more than 1M files when using archivemount. Ratarmount 0.10.0 improves index creation upon earlier versions by batching SQLite insertions.

Cons

• Unfortunately, gzip compressed TAR files are roughly one order of magnitude slower with ratarmount than archivemount during first time mounting. But, this slowdown will be amortized after the second subsequent mount thanks to the index file.
• Getting a lot of metadata for archive contents as demonstrated by calling find on the mount point is an order of magnitude slower compared to archivemount. Because the C-based fuse-archive is even slower than ratarmount, the difference is very likely that archivemount uses the low-level FUSE interface while ratarmount and fuse-archive use the high-level FUSE interface.

For more in-depth benchmarks, see the Github page.

Answer 2

You could also create a compressed squashfs image

mksquashfs /etc squashfs.img -comp xz
mkdir img
mount -o squashfs,ro squashfs.img img

In order to do this you'll need to extract your tar.gz archvie.

The advantage is also that the image has better fault tolerancy than gz.

Answer 3

The problem here is with the format, the TAR (Tape ARchive) format is designed for sequential access, not random access. And gzip is a good complement to tar, since it is a stream based compression format, also not for random access.

So a high level tool that does not interact with the compressed blocks directly, will have to parse through the entire file every time it needs to read anything, first to get you the list of files, then perhaps the cache invalidates and it reads it again, and then for each file you copy off it might read through it again. You can make a tool that remembers the position of each file, and what blocks it needs to decompress to get it, but it seems that few have bothered with that.

If you want this to go faster, do a tar tzf file.tar.gz > filelist, open that file list in vim, gedit or whatever, remove the lines of files you do not need, save, and then extract them with tar xzf file.tar.gz -T filelist -C extracted/.

To get random access to a compressed file, you should use perhaps zip with posix extensions, rar, or as dru8274 suggested, squashfs, or even ZFS with compression turned on, or btrfs if btrfs has gotten compression to work at the time of reading.

Answer 4

If you only need read-only access, check Fuse-Archive. readdir might not be that faster than archivemount, but for some workflows it is.

Answer 5

This will not cover all use-cases as it restricts use to a text-editor. But, if you only care about read-access, you might find this helpful for some situations. vim, when run on a tarball will show you the content hierarchy of the archive (similar to how it will display a file hierarchy if run on a directory). By selecting one of the files in the list, it will open the selected file in a read-only buffer.

Again, this doesn't necessarily offer access to images or other media, but if all you need is to see the contents or access text-based files only, then this should be helpful.

Note: this will not work on all archive formats.

Answer 6

My approach. If you have enough free disk space on an external USB drive or external/secondary HDD drive with enough space, then consider just extracting your .tar.gz file. Thinking you probably don't want 3 million files on your main system disk, as that could slow things down. I'd recommend that the external disk in this case have a filesystem that handles a huge number of files easily: thinking ReiserFS, ext4 (with dir_index option), XFS, maybe BtrFS. It could might take 1-2 hours to do the extract, but you could just go get lunch in the meantime or let it run overnight; when you come back, accessing the extracted files should be performant.