16

At the moment I am using ArchiveMount to mount a 123,000 kb archive that contains more then 3 million files inside. So far it has been mounting for 5+ hours and still isn't finished.

Is there a better way to mount a .tar.gz file? I am trying to mount to a folder, and uncompressed it takes a few gigs. I don't even need write mode, just read-only is sufficient.

  • There's also AVFS; I have no idea if it'll perform better. – Gilles 'SO- stop being evil' Nov 6 '11 at 16:25
  • 8
    If your files were compressed as a squashfs module instead of as a tarball, then read-only access would be very quick - you just (loop) mount the squashfs module. Requires the squashfs-tools package. – dru8274 Nov 7 '11 at 6:05
  • I'm currently programming such a file system. Wait a couple of months and it's going to be there. – FUZxxl Feb 16 '16 at 21:50
  • @FUZxxl Well, its been 2 years, did you ever write this utility? – cybernard Feb 20 '19 at 18:00
  • @cybernard FUSE frustrated me so much that I gave up on this project. I hate this undocumented piece of shit. I do keep this on the back burner and might take it back on later. – FUZxxl Feb 20 '19 at 20:10
8

I wrote a faster alternative ratarmount, which "works for me", because this problem kept bugging me.

You can use it like this:

pip3 install --user ratarmount
ratarmount my-huge-tar.tar mount-folder
ls -la mount-folder # will show the contents of the tar top-level

When you are done you can unmount it like any FUSE mount:

fusermount -u mount-folder

Why is it faster than archivemount?

It depends on what you measure.

Here is a benchmark of memory footprint and required time for first mounting, as well as access times for a simple cat <file-in-tar> command and a simple find command.

Benchmark comparison between ratarmount and archivemount

Folders containing each 1k files were created and the number of folders is varied.

The lower left plot shows error bars indicating the minimum and maximum measured times for cat <file> for 10 randomly chosen files.

File seek time

The killer comparison is the time it takes for cat <file> to finish. For some reason, this scales linearly with the TAR file size (approx. bytes per file x number of files) for archivemount while being of constant time in ratarmount. This makes it look like archivemount does not even support seeking at all.

For compressed TAR files, this is especially noticeable. cat <file> takes more than twice as long as mounting the whole .tar.bz2 file! For example, the TAR with 10k empty(!) files takes 2.9s to mount with archivemount but depending on the file which is accessed, the access with cat takes between 3ms and 5s. The time it takes seems to depend on the position of the file inside the TAR. Files at the end of the TAR take longer to seek to; indicating that the "seek" is emulated and all the contents in the TAR before the file are being read.

That getting the file contents can take more than twice as much time as mounting the whole TAR is unexpected in on itself. At least, it should finish in the same amount of time as mounting. One explanation would be that the file is being emulatedly seeked to more than once, maybe even thrice.

Ratarmount seemingly takes always the same amount of time to get a file because it supports true seeking. For bzip2 compressed TARs, it even seeks to the bzip2 block, whose addresses are also stored in the index file. Theoretically, the only part which should scale with the number of files is the lookup in the index and that should scale with O(log(n)) because it is sorted by file path and name.

Memory footprint

In general, if you have more than 20k files inside the TAR, then the memory footprint of ratarmount will be smaller because the index is written to disk as it is created and therefore has a constant memory footprint of roughly 30MB on my system.

A small exception is the gzip decoder backend, which for some reason requires more memories as the gzip gets larger. This memory overhead might be the index required for seeking inside the TAR but further investigation is needed as I did not write that backend.

In contrast, archivemount keeps the whole index, which is, e.g., 4GB for 2M files, completely in memory for as long as the TAR is mounted.

Mounting time

My favorite feature is ratarmount being able to mount the TAR without noticeably delay on any subsequent try. This is because the index, which maps filenames to metadata and the position inside the TAR, is written to an index file created next to the TAR file.

The required time for mounting behaves kinda weird in archivemount. Starting from roughly 20k files it begins to scale quadratically instead of linearly with respect to the number of files. This means that starting from roughly 4M files, ratarmount begins to be much faster than archivemount even though for smaller TAR files it's up to 10 times slower! Then again, for smaller files, it does not matter much whether it takes 1s or 0.1s to mount the tar (the first time).

The mounting times for bz2 compressed files are the most comparable at all times. This is very likely because it is bound by the speed of the bz2 decoder. Ratarmount is roughly 2x slower here. I hope to make ratarmount the clear winner by parallelizing the bz2 decoder in the near future, which even for my 8-year-old system could yield a 4x speedup.

Time to get metadata

When simply listing all files with find inside the TAR (find also seems to call stat for each file!?), ratarmount is 10x slower than archivemount for all tested cases. I hope to improve upon this in the future. But currently, it looks like a design problem because of using Python and SQLite instead of a pure C program.

| improve this answer | |
  • How would the OP install and use this to solve their problem? – Jeff Schaller Feb 24 '19 at 14:19
  • @JeffSchaller I added the install instructions from the github readme.md – mxmlnkn Feb 25 '19 at 8:56
  • ratarmount - good idea, but useless because of Python 2/3. But if you publish release package with static build of Python and its dependences, then it will be usefull. – user1742529 Apr 17 at 9:32
  • @user1742529 You want to use ratarmount on system without any Python installation on it? Python is quite ubiquitous, it doesn't hurt having it installed. I could look into something like building an AppImage, FlatPak, or Snap, would that suffice in your opinion? Not sure if they work for this use case though. – mxmlnkn Apr 17 at 9:42
  • @mxmlnkn yes, on system without any Python installation. you can pack Python+Modules+Ratarmount with something simple like makeself github.com/megastep/makeself , then make installation e.g. to /opt with one single command. – user1742529 Apr 22 at 9:19
7

You could also create a compressed squashfs image

mksquashfs /etc squashfs.img -comp xz
mkdir img
mount -o squashfs,ro squashfs.img img

In order to do this you'll need to extract your tar.gz archvie.

The advantage is also that the image has better fault tolerancy than gz.

| improve this answer | |
6

The problem here is with the format, the TAR (Tape ARchive) format is designed for sequential access, not random access. And gzip is a good complement to tar, since it is a stream based compression format, also not for random access.

So a high level tool that does not interact with the compressed blocks directly, will have to parse through the entire file every time it needs to read anything, first to get you the list of files, then perhaps the cache invalidates and it reads it again, and then for each file you copy off it might read through it again. You can make a tool that remembers the position of each file, and what blocks it needs to decompress to get it, but it seems that few have bothered with that.

If you want this to go faster, do a tar tzf file.tar.gz > filelist, open that file list in vim, gedit or whatever, remove the lines of files you do not need, save, and then extract them with tar xzf file.tar.gz -T filelist -C extracted/.

To get random access to a compressed file, you should use perhaps zip with posix extensions, rar, or as dru8274 suggested, squashfs, or even ZFS with compression turned on, or btrfs if btrfs has gotten compression to work at the time of reading.

| improve this answer | |
  • 3
    To get random access to a compressed file, you could also use pixz. – kubanczyk Jun 29 '14 at 20:42
0

This will not cover all use-cases as it restricts use to a text-editor. But, if you only care about read-access, you might find this helpful for some situations. vim, when run on a tarball will show you the content hierarchy of the archive (similar to how it will display a file hierarchy if run on a directory). By selecting one of the files in the list, it will open the selected file in a read-only buffer.

Again, this doesn't necessarily offer access to images or other media, but if all you need is to see the contents or access text-based files only, then this should be helpful.

Note: this will not work on all archive formats.

| improve this answer | |
  • vim's built-in archive viewer still needs to scan through the whole file to obtain a listing, hardly faster than avfs and archivemount. and displaying such a huge listing of millions of lines is also terrible. – 把友情留在无盐 Jul 25 '16 at 6:06
0

My approach. If you have enough free disk space on an external USB drive or external/secondary HDD drive with enough space, then consider just extracting your .tar.gz file. Thinking you probably don't want 3 million files on your main system disk, as that could slow things down. I'd recommend that the external disk in this case have a filesystem that handles a huge number of files easily: thinking ReiserFS, ext4 (with dir_index option), XFS, maybe BtrFS. It could might take 1-2 hours to do the extract, but you could just go get lunch in the meantime or let it run overnight; when you come back, accessing the extracted files should be performant.

| improve this answer | |
  • there is no need for an additional media, a loop device is sufficient. – 把友情留在无盐 Jul 25 '16 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.