2

I have a directory called Pages of 2.2 million HTML files (about 80 GB) on an Ubuntu server. I compressed it with 7-Zip using this command:

7z a -mx=9 Pages.7z Pages

It took around 5-6 hours to compress (seems excessive). Compressed size is about 2.3 GB.

I then downloaded it to my main computer (Ubuntu, Intel® Xeon® CPU E5-1650 v2 @ 3.50GHz). Every time I try to extract, it starts off at disappointing, but acceptable speed, but slows down to a crawl as it gets further along (ran overnight and when I woke up it was doing about 300 files per minute).

However, on my Windows machine (Intel® Xeon® CPU E5-2687W @ 3.10GHz 3.10 GHz, which is only a slightly better machine, I extracted the entire directory in 15-20 minutes. It also clearly made use of multiple processors, which I can't get 7-Zip to do on Ubuntu.

Obviously I can't have an extraction take several days, nor should I.

My sense is this has to do with something I don't know about Ubuntu (I'm a recovering Windows user) or my file system rather than 7-Zip. Any help would be tremendously appreciated.

My main computer uses ext4 file system, and the version of 7-Zip I have is 9.20:

7-Zip [64] 9.20 p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,12 CPUs)

Update:

I should clarify that I actually have one drive on my main Ubuntu installation that is ext4 (my ssd), though I have another one that is ntfs (I think I remember this being recommended by Ubuntu during installation, perhaps b/c I set it up as a raid array). The problem of slowing down over time was happening regardless of which I was working from.

Following advice in the comments, I used my Windows machine to unzip the compressed file, restructure the directory with 4096 subdirectories, and re-zip it (though this time I used the default compression level rather than maximum, and specified lzma2). I then transferred it to my Ubuntu machine (the ext4 SSD specifically) and unzipped. It worked perfectly as I would expect - very fast.

However, as another commenter noted, part of the problem here is likely just that my drives on the Ubuntu machine are not indexed (they are on Windows), and I might not have to restructure directories at all if I do index (which I've been wanting to do anyway). I'm currently trying to figure out how to do that successfully and safely...and will report back with any useful results.

I've also tried restructuring a directory already on my Ubuntu machine using python, which is going unreasonably slow. Perhaps it's a python issue rather than Linux/ext4/ntfs or perhaps it also has to do with indexing, or perhaps it is b/c the source directory has 2.2 million files in one directory...:

for fileName in series:
    if not os.path.exists('[...]/Pages2/' + fileName[:3] + '/' + fileName):
        shutil.copy('[...]/Pages/' + fileName, '[...]/Pages2/' + fileName[:3] + '/' + fileName)

2 Answers 2

3

I finally figured out the actual answer when I read the wikipedia entry for XZ (https://en.wikipedia.org/wiki/Xz):

One can think of xz as a stripped-down version of the 7-Zip program. xz has its own file format rather than the .7z format used by 7-Zip (which lacks support for Unix-like file system metadata[2]).

It is in fact okay to have millions of small files in a single directory, it would seem, on either NTFS or EXT-4 with Ubuntu (perhaps not advisable for other reasons however). There was also nothing wrong with the indexing on my file systems. The reason 7zip slows down when trying to extract a massive directory has everything to do with the writers of 7zip not caring much about Linux/Unix users.

This does half make me wonder whether whoever wrote Nautilus is similarly contemptuous of Linux users...b/c it really doesn't like directories with lots of files either, whereas Windows Explorer has no problems with it.

1
  • Unix-like file system metadata is the Unix permission rwx and has nothing to do with the speed. The 7z format is designed without Unix permission in mind, but that has no relation whatsoever with the p7zip program being slow
    – phuclv
    Oct 21, 2022 at 17:28
1

are you talking about one directory with 2.2mil files? Ext Filesystems like to get slow when you work with that many files.

6
  • 3
    Correct, if they are not in various subdirectories, ext(2/3/4) filesystems will come to a crawl when you get to 100.000+ files in a directory. On XFS filesystems this will not happen that quickly, but it's never a good idea to have that many files in a single directory.
    – Sig-IO
    Jun 18, 2015 at 22:17
  • I see. That's odd though b/c I do have several directories with over a million files in them on my Ubuntu machine and haven't had too much trouble with them when moving files around. Also, the server I compressed the file with is using ext4, and the file compressed okay. I can try unzipping on my windows machine then rezipping after adding some structure to the directory. Does Windows just have a far superior file system or something?
    – KCzar
    Jun 18, 2015 at 22:25
  • 1
    As @Sig-IO mentioned, this is an effect of many files in one directory, just because the file system isn't optimized for such a usage. If you distribute your files over several directories, it will act as fast as you used to know it.
    – ikrabbe
    Jun 18, 2015 at 22:27
  • Are you guys sure this is accurate? Check this: en.wikipedia.org/wiki/HTree " HTree indexes are used in the ext3 and ext4 Linux filesystems, and were incorporated into the Linux kernel around 2.5.40.[2] HTree indexing improved the scalability of Linux ext2 based filesystems from a practical limit of a few thousand files, into the range of tens of millions of files per directory."
    – KCzar
    Jun 19, 2015 at 1:18
  • 2
    @KCzar that's correct, but it needs to be explicitly enabled on the filesystem (it's the dir_index feature). If you have the rights to do so you can check whether it's enabled using dumpe2fs. Jun 19, 2015 at 4:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .