I wrote a faster alternative ratarmount, which "works for me", because this problem kept bugging me.
You can use it like this:
pip3 install --user ratarmount
ratarmount my-huge-tar.tar mount-folder
ls -la mount-folder # will show the contents of the tar top-level
When you are done you can unmount it like any FUSE mount:
fusermount -u mount-folder
Why is it faster than archivemount?
It depends on what you measure.
Here is a benchmark of memory footprint and required time for first mounting, as well as access times for a simple
cat <file-in-tar> command and a simple
Folders containing each 1k files were created and the number of folders is varied.
The lower left plot shows error bars indicating the minimum and maximum measured times for
cat <file> for 10 randomly chosen files.
File seek time
The killer comparison is the time it takes for
cat <file> to finish.
For some reason, this scales linearly with the TAR file size (approx. bytes per file x number of files) for archivemount while being of constant time in ratarmount.
This makes it look like archivemount does not even support seeking at all.
For compressed TAR files, this is especially noticeable.
cat <file> takes more than twice as long as mounting the whole .tar.bz2 file!
For example, the TAR with 10k empty(!) files takes 2.9s to mount with archivemount but depending on the file which is accessed, the access with
cat takes between 3ms and 5s.
The time it takes seems to depend on the position of the file inside the TAR.
Files at the end of the TAR take longer to seek to; indicating that the "seek" is emulated and all the contents in the TAR before the file are being read.
That getting the file contents can take more than twice as much time as mounting the whole TAR is unexpected in on itself.
At least, it should finish in the same amount of time as mounting.
One explanation would be that the file is being emulatedly seeked to more than once, maybe even thrice.
Ratarmount seemingly takes always the same amount of time to get a file because it supports true seeking.
For bzip2 compressed TARs, it even seeks to the bzip2 block, whose addresses are also stored in the index file.
Theoretically, the only part which should scale with the number of files is the lookup in the index and that should scale with O(log(n)) because it is sorted by file path and name.
In general, if you have more than 20k files inside the TAR, then the memory footprint of ratarmount will be smaller because the index is written to disk as it is created and therefore has a constant memory footprint of roughly 30MB on my system.
A small exception is the gzip decoder backend, which for some reason requires more memories as the gzip gets larger.
This memory overhead might be the index required for seeking inside the TAR but further investigation is needed as I did not write that backend.
In contrast, archivemount keeps the whole index, which is, e.g., 4GB for 2M files, completely in memory for as long as the TAR is mounted.
My favorite feature is ratarmount being able to mount the TAR without noticeably delay on any subsequent try.
This is because the index, which maps filenames to metadata and the position inside the TAR, is written to an index file created next to the TAR file.
The required time for mounting behaves kinda weird in archivemount.
Starting from roughly 20k files it begins to scale quadratically instead of linearly with respect to the number of files.
This means that starting from roughly 4M files, ratarmount begins to be much faster than archivemount even though for smaller TAR files it's up to 10 times slower!
Then again, for smaller files, it does not matter much whether it takes 1s or 0.1s to mount the tar (the first time).
The mounting times for bz2 compressed files are the most comparable at all times.
This is very likely because it is bound by the speed of the bz2 decoder.
Ratarmount is roughly 2x slower here.
I hope to make ratarmount the clear winner by parallelizing the bz2 decoder in the near future, which even for my 8-year-old system could yield a 4x speedup.
Time to get metadata
When simply listing all files with
find inside the TAR (find also seems to call stat for each file!?), ratarmount is 10x slower than archivemount for all tested cases.
I hope to improve upon this in the future.
But currently, it looks like a design problem because of using Python and SQLite instead of a pure C program.