I have a directory called
Pages of 2.2 million HTML files (about 80 GB) on an Ubuntu server. I compressed it with 7-Zip using this command:
7z a -mx=9 Pages.7z Pages
It took around 5-6 hours to compress (seems excessive). Compressed size is about 2.3 GB.
I then downloaded it to my main computer (Ubuntu, Intel® Xeon® CPU E5-1650 v2 @ 3.50GHz). Every time I try to extract, it starts off at disappointing, but acceptable speed, but slows down to a crawl as it gets further along (ran overnight and when I woke up it was doing about 300 files per minute).
However, on my Windows machine (Intel® Xeon® CPU E5-2687W @ 3.10GHz 3.10 GHz, which is only a slightly better machine, I extracted the entire directory in 15-20 minutes. It also clearly made use of multiple processors, which I can't get 7-Zip to do on Ubuntu.
Obviously I can't have an extraction take several days, nor should I.
My sense is this has to do with something I don't know about Ubuntu (I'm a recovering Windows user) or my file system rather than 7-Zip. Any help would be tremendously appreciated.
My main computer uses ext4 file system, and the version of 7-Zip I have is 9.20:
7-Zip  9.20 p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,12 CPUs)
I should clarify that I actually have one drive on my main Ubuntu installation that is ext4 (my ssd), though I have another one that is ntfs (I think I remember this being recommended by Ubuntu during installation, perhaps b/c I set it up as a raid array). The problem of slowing down over time was happening regardless of which I was working from.
Following advice in the comments, I used my Windows machine to unzip the compressed file, restructure the directory with 4096 subdirectories, and re-zip it (though this time I used the default compression level rather than maximum, and specified lzma2). I then transferred it to my Ubuntu machine (the ext4 SSD specifically) and unzipped. It worked perfectly as I would expect - very fast.
However, as another commenter noted, part of the problem here is likely just that my drives on the Ubuntu machine are not indexed (they are on Windows), and I might not have to restructure directories at all if I do index (which I've been wanting to do anyway). I'm currently trying to figure out how to do that successfully and safely...and will report back with any useful results.
I've also tried restructuring a directory already on my Ubuntu machine using python, which is going unreasonably slow. Perhaps it's a python issue rather than Linux/ext4/ntfs or perhaps it also has to do with indexing, or perhaps it is b/c the source directory has 2.2 million files in one directory...:
for fileName in series: if not os.path.exists('[...]/Pages2/' + fileName[:3] + '/' + fileName): shutil.copy('[...]/Pages/' + fileName, '[...]/Pages2/' + fileName[:3] + '/' + fileName)