1

I have a directory called Pages of 2.2 million HTML files (about 80 GB) on an Ubuntu server. I compressed it with 7-Zip using this command:

7z a -mx=9 Pages.7z Pages

It took around 5-6 hours to compress (seems excessive). Compressed size is about 2.3 GB.

I then downloaded it to my main computer (Ubuntu, Intel® Xeon® CPU E5-1650 v2 @ 3.50GHz). Every time I try to extract, it starts off at disappointing, but acceptable speed, but slows down to a crawl as it gets further along (ran overnight and when I woke up it was doing about 300 files per minute).

However, on my Windows machine (Intel® Xeon® CPU E5-2687W @ 3.10GHz 3.10 GHz, which is only a slightly better machine, I extracted the entire directory in 15-20 minutes. It also clearly made use of multiple processors, which I can't get 7-Zip to do on Ubuntu.

Obviously I can't have an extraction take several days, nor should I.

My sense is this has to do with something I don't know about Ubuntu (I'm a recovering Windows user) or my file system rather than 7-Zip. Any help would be tremendously appreciated.

My main computer uses ext4 file system, and the version of 7-Zip I have is 9.20:

7-Zip [64] 9.20 p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,12 CPUs)

Update:

I should clarify that I actually have one drive on my main Ubuntu installation that is ext4 (my ssd), though I have another one that is ntfs (I think I remember this being recommended by Ubuntu during installation, perhaps b/c I set it up as a raid array). The problem of slowing down over time was happening regardless of which I was working from.

Following advice in the comments, I used my Windows machine to unzip the compressed file, restructure the directory with 4096 subdirectories, and re-zip it (though this time I used the default compression level rather than maximum, and specified lzma2). I then transferred it to my Ubuntu machine (the ext4 SSD specifically) and unzipped. It worked perfectly as I would expect - very fast.

However, as another commenter noted, part of the problem here is likely just that my drives on the Ubuntu machine are not indexed (they are on Windows), and I might not have to restructure directories at all if I do index (which I've been wanting to do anyway). I'm currently trying to figure out how to do that successfully and safely...and will report back with any useful results.

I've also tried restructuring a directory already on my Ubuntu machine using python, which is going unreasonably slow. Perhaps it's a python issue rather than Linux/ext4/ntfs or perhaps it also has to do with indexing, or perhaps it is b/c the source directory has 2.2 million files in one directory...:

for fileName in series:
    if not os.path.exists('[...]/Pages2/' + fileName[:3] + '/' + fileName):
        shutil.copy('[...]/Pages/' + fileName, '[...]/Pages2/' + fileName[:3] + '/' + fileName)
2

I finally figured out the actual answer when I read the wikipedia entry for XZ (https://en.wikipedia.org/wiki/Xz):

One can think of xz as a stripped-down version of the 7-Zip program. xz has its own file format rather than the .7z format used by 7-Zip (which lacks support for Unix-like file system metadata[2]).

It is in fact okay to have millions of small files in a single directory, it would seem, on either NTFS or EXT-4 with Ubuntu (perhaps not advisable for other reasons however). There was also nothing wrong with the indexing on my file systems. The reason 7zip slows down when trying to extract a massive directory has everything to do with the writers of 7zip not caring much about Linux/Unix users.

This does half make me wonder whether whoever wrote Nautilus is similarly contemptuous of Linux users...b/c it really doesn't like directories with lots of files either, whereas Windows Explorer has no problems with it.

1

are you talking about one directory with 2.2mil files? Ext Filesystems like to get slow when you work with that many files.

  • 3
    Correct, if they are not in various subdirectories, ext(2/3/4) filesystems will come to a crawl when you get to 100.000+ files in a directory. On XFS filesystems this will not happen that quickly, but it's never a good idea to have that many files in a single directory. – Sig-IO Jun 18 '15 at 22:17
  • I see. That's odd though b/c I do have several directories with over a million files in them on my Ubuntu machine and haven't had too much trouble with them when moving files around. Also, the server I compressed the file with is using ext4, and the file compressed okay. I can try unzipping on my windows machine then rezipping after adding some structure to the directory. Does Windows just have a far superior file system or something? – KCzar Jun 18 '15 at 22:25
  • 1
    As @Sig-IO mentioned, this is an effect of many files in one directory, just because the file system isn't optimized for such a usage. If you distribute your files over several directories, it will act as fast as you used to know it. – ikrabbe Jun 18 '15 at 22:27
  • Are you guys sure this is accurate? Check this: en.wikipedia.org/wiki/HTree " HTree indexes are used in the ext3 and ext4 Linux filesystems, and were incorporated into the Linux kernel around 2.5.40.[2] HTree indexing improved the scalability of Linux ext2 based filesystems from a practical limit of a few thousand files, into the range of tens of millions of files per directory." – KCzar Jun 19 '15 at 1:18
  • 2
    @KCzar that's correct, but it needs to be explicitly enabled on the filesystem (it's the dir_index feature). If you have the rights to do so you can check whether it's enabled using dumpe2fs. – Stephen Kitt Jun 19 '15 at 4:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.