I am running
unzip to unzip huge files. However, my cpu usage is under 15 percent and ram is only utilizing 1-1.2 GB out of 8 GB.
Is there a way to allocate more cpu power and ram to this
I am on Lubuntu 16.04
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
Programs take all the memory and CPU power they can get, unless they have built-in limitations.
unzip has no such built-in limitations. You could give it less, but you can't give it more, because by default it's allowed to take as much as it wants.
Unzipping is not a memory-intensive process. The main memory cost of unzipping a huge archive is that unzip keeps the list of files in memory.
The limiting factor for speed may be CPU power or disk (or the network if you're reading or writing a file to the network). It depends how fast your disk is relative to your CPU. Check whether the process is taking 100% of one core. If it isn't, then the only way to speed it up would be to speed up the input/output. This can mean a faster disk, or arranging to put the input and the output on separate disks.
If the process is taking 100% of one core, then you can speed it up by parallelizing. For many compression formats, decompression of one file is inherently non-parallelizable, because the format is very adaptative: compression is achieved by looking for repeated patterns and replacing them by some indirect reference to a previous pattern. Some compression formats have “reinitialization points” that allow decompressing each block independently; I know this is as least the case for bzip2. Some compression tools do this even if the format doesn't require it. But as far as I know, this is not the case for zip. On the other hand, zip compresses each member of an archive separately, so it's possible to decompress each file independently. Thus, if you have n cores, you can keep all of them busy decompressing separate files (if your I/O is up to speed).
The problem is then to find a parallel unzip implementation. I think p7zip supports it with
7z x -mmt=on foo.zip or
7z x -mmt=8 (to use 8 cores), but p7zip's documentation is not very good and I haven't confirmed that this does parallelize.
(since I cannot add a comment to another answer)
htop is one of the tools to see process info. If the 'S' column (for Status) is showing a 'D', it means that the process is blocked, usually waiting for I/O operations to finish. Increase bandwith for network or storage (eg RAID or faster HDD/SSD/NVMe) to deal with that bottleneck.
iotop is another useful tool to detect such bottlenecks; it would show you 100% I/O usage for that process.
You can use pigz which does parallel compress/decompress using all CPU cores, the decompression unpigz was serial at one point but I think it was enhanced. It really super cool