2

I am running unzip to unzip huge files. However, my cpu usage is under 15 percent and ram is only utilizing 1-1.2 GB out of 8 GB.

Is there a way to allocate more cpu power and ram to this unzip program?

Thank you.

I am on Lubuntu 16.04

2
  • Sounds like either you have 6 cores and it maxxes out on one of them or your disk cannot read/write data any faster. – Jonas Berlin Aug 9 '19 at 20:04
  • 4 cores I have. – Faraz Durrani Aug 9 '19 at 21:08
2

Programs take all the memory and CPU power they can get, unless they have built-in limitations. unzip has no such built-in limitations. You could give it less, but you can't give it more, because by default it's allowed to take as much as it wants.

Unzipping is not a memory-intensive process. The main memory cost of unzipping a huge archive is that unzip keeps the list of files in memory.

The limiting factor for speed may be CPU power or disk (or the network if you're reading or writing a file to the network). It depends how fast your disk is relative to your CPU. Check whether the process is taking 100% of one core. If it isn't, then the only way to speed it up would be to speed up the input/output. This can mean a faster disk, or arranging to put the input and the output on separate disks.

If the process is taking 100% of one core, then you can speed it up by parallelizing. For many compression formats, decompression of one file is inherently non-parallelizable, because the format is very adaptative: compression is achieved by looking for repeated patterns and replacing them by some indirect reference to a previous pattern. Some compression formats have “reinitialization points” that allow decompressing each block independently; I know this is as least the case for bzip2. Some compression tools do this even if the format doesn't require it. But as far as I know, this is not the case for zip. On the other hand, zip compresses each member of an archive separately, so it's possible to decompress each file independently. Thus, if you have n cores, you can keep all of them busy decompressing separate files (if your I/O is up to speed).

The problem is then to find a parallel unzip implementation. I think p7zip supports it with 7z x -mmt=on foo.zip or 7z x -mmt=8 (to use 8 cores), but p7zip's documentation is not very good and I haven't confirmed that this does parallelize.

1

(since I cannot add a comment to another answer)
htop is one of the tools to see process info. If the 'S' column (for Status) is showing a 'D', it means that the process is blocked, usually waiting for I/O operations to finish. Increase bandwith for network or storage (eg RAID or faster HDD/SSD/NVMe) to deal with that bottleneck.

iotop is another useful tool to detect such bottlenecks; it would show you 100% I/O usage for that process.

0
1

You can use pigz which does parallel compress/decompress using all CPU cores, the decompression unpigz was serial at one point but I think it was enhanced. It really super cool

https://linux.die.net/man/1/pigz

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.