16

We have 100+ GB files on a Linux machine, and while trying to perform gzip using below command, gzip is taking minimum 1-2 hours to complete:

gzip file.txt

Is there a way we can make gzip to run fast with the same level of compression happening when we use gzip?


CPU: Intel(R) Core(TM) i3-2350M CPU @2.30 GHz

23
  • 6
    With the bandwidth available to me in my Fiber@Home connection, sending 100 Gb over the internet would've only taken me 2-3 hours. What exactly are your goals? A compressed archive of this size is nearly useless, since accessing the content implies unzipping the entire thing. You may want to re-think your use case and maybe just buy a bigger hard drive.
    – Nelson
    Commented Dec 11, 2020 at 6:33
  • 7
    Are you limited to gzip? If not, parallel bzip2 (pbzip2) and parallel xz (xz --threads) exist.
    – RonJohn
    Commented Dec 11, 2020 at 7:09
  • 5
    @Ravi xz --threads means to use the xz command with the --threads parameter to paralellize the processing. Commented Dec 11, 2020 at 11:39
  • 6
    Note that if you are using a single hard disk drive (as opposed to SSDs or RAID setups), the I/O alone will probably take a significant part of the time, so even parallelising may not gain that much.
    – jcaron
    Commented Dec 11, 2020 at 13:30
  • 3
    @RonJohn: zstdmt --adapt aims to hit that sweet spot that saturates I/O and CPU, adapting on the fly (within a set range) to "perceived I/O conditions". man page. IDK if --adapt is usable with --format=gzip, to take advantage of that and zstd's threaded I/O (and compression) for the .gz format. Commented Dec 14, 2020 at 18:19

8 Answers 8

31

If you are using gzip, you use mostly one processor core (well, some parts of the task, like reading and writing data are kernel tasks and kernel will use another core). Have a look at some multicore-capable gzip replacements, e.g. MiGz (https://github.com/linkedin/migz) or Pigz (https://zlib.net/pigz/, for some longer explanation see also e.g. https://medium.com/ngs-sh/pigz-a-faster-alternative-to-gzip-for-big-files-d5909e46d659).

6
  • 5
    Since you have multiple files you could instead gzip more than one at the same time Commented Dec 11, 2020 at 16:29
  • 6
    pigz is pretty universally available. Try to install it with your package manager of choice.
    – Rich
    Commented Dec 11, 2020 at 16:29
  • pigz is probably already installed on your system. Commented Dec 12, 2020 at 4:27
  • @MichaelHampton .. I ran man command but i cant see this available -- No manual entry for pigz
    – Ravi
    Commented Dec 13, 2020 at 17:23
  • 1
    man is just a man page. which pigz would be a bit more related to a command availability...
    – d.c.
    Commented Dec 13, 2020 at 21:28
20

We have 100+ GB files and while trying to perform gzip using below command , gzip is taking minimum 1-2 hours to get complete

With CPU (taken from a comment): Intel® Core™ i3-2350M @ 2.30GHz, which has:

Number of Cores: 2; Number of Threads: 4


Your CPU sounds like a bottleneck according to this low score (benchmark), also to note this is a laptop CPU, quite an old one. In this setup, I expect a classic HDD instead of some modern SSD too, together with possibly low RAM, etc.


The conclusion is possibly no, you cannot do software-wise anything to get the higher performance of gzip on your computer without getting a lower compression ratio, of course.


Default compression setting is -6 if I am not mistaken, you could hit for example -2:

gzip -2 file.txt

and compare the results yourself. See the manual page for more settings.


UPDATE on pigz

Today, 2021-Jun-03, I myself needed to compress a rather large file of size 256 GB (239 GiB), and I somewhat tested gzip, bzip2, xz, and I found all of these being unable to take full advantage of my CPU (i7-7700HQ) and being fast, which is our goal in this Q&A.

In the end, I downloaded pigz (man page) from its home page, and compiled it simply by running make, then as I did not prefer to directly put it onto my PATH, so I created a Bash alias to the binary.

It might be useful to note how to watch the (possibly long) progress:

Example #1 (reading a prepared disk image and writing gzip'ed file in the same directory):

file=disk.img; pv < "$file" | pigz -2 > "$file".gz

Example #2 (reading disk directly and writing gzip'ed file in the the current directory):

dev=/dev/nvme0n1; file=disk.img.gz; pv < "$dev" | pigz -9 > "$file"

Conclusion

I now recommend using pigz, the parallel implementation of gzip, on very large files.

4
  • Regarding the 2021-Jun-03 update - have you tried xz -T0 to utilize all CPU cores?
    – orip
    Commented Jun 10, 2021 at 8:18
  • 1
    @orip Yes, I did. However, only a small portion of the normal xz is able to utilize all threads. + It is painfully slow, h*ll, extremely slow, depending on settings; pixz would be of more service (man page) Commented Jun 10, 2021 at 9:37
  • Have you tried zstd? It's much faster than xz and outperforms gzip in both time and compression size. gregoryszorc.com/blog/2017/03/07/… Commented Jun 5 at 17:52
  • @SurpriseDog I highly doubt any algorithm could be faster than pigz, but after I am not sick anymore, I will try pzstd for good measure. Thanks. Commented Jun 5 at 20:18
10

Do you specifically need gzip, or would other compression algorithms be an option? zstandard and lzop are both significantly faster than gzip.

4
  • 2
    lz4 has a somewhat better tradeoff of compression ratio to performance than lzop. (both are about the same speed at -2, but lzop makes a slightly smaller file.) It's used for stuff like transparent filesystem compression in btrfs. But don't turn up the compression ratio unless you need the blazing fast decompression: if you don't need the blazing fast decompression of lzop or lz4, lz4 -4 is much slower, similar to gzip -4 which compresses somewhat better. And pigz -4 for parallel gzip lets you use both cores for a 2x speedup. (lzop -4 or -6 doesn't improve like lz4 does) Commented Dec 12, 2020 at 2:29
  • 4
    @PeterCordes btw btrfs has switched from lzo to zstandard by default
    – hanshenrik
    Commented Dec 12, 2020 at 16:16
  • 4
    community.centminmod.com/threads/… benchmarks compression speed and size on a Haswell quad-core for various compressors, including pigz vs. zstd vs. parallel bzip2 vs. parallel lzma. No lz4 or lzop in that test, though. Commented Dec 14, 2020 at 17:58
  • @PeterCordes I found that lzop was great when I needed to migrate an on-prem Oracle database to AWS Postgres.
    – RonJohn
    Commented Jul 20, 2022 at 17:32
8

As the others have pointed out, gzip is single-threaded.

If you have multiple files to process, you could do that in parallel:

find -type f -not -name '*.gz' -print0 | xargs -tr0n 1 -P$(nproc) gzip
  • find: scan the file system tree
  • -type f: filter for regular files
  • -not: invert the next test
    • -name '*.gz': any files that are already compressed
  • print0: print the name to stdout, followed by a NUL byte
  • | pipe into
  • xargs: read elements from stdin, and pass them as arguments to another program
  • -t: write the commands run to the console
  • -r: do not execute any command if no elements are present
  • -0: expect elements to be separated by NUL bytes
  • -n 1: give one element to each invocation
  • -P ...: run as many commands in parallel
    • $(...): run this command and substitute its output
      • nproc: get the number of processors
  • gzip: the command to be run for each file

Note that gzip is still rather fast as an algorithm, so you are likely to be I/O bound in this, and at the same time the compression isn't that great. If you are free to choose another compression method, you can use xz instead, which compresses way better, but needs more CPU time for that.

In theory, xz can parallelize internally, but that gives slightly worse compression:

xz -T$(nproc) *.txt
3
  • 1
    compressing files separately will result in worse compression ratio because you can't have solid archives anymore
    – phuclv
    Commented Dec 12, 2020 at 2:53
  • 3
    The CPU doesn't care if you launch 4 instances of gzip, but the disk sure does. Compressing >1 files at once will cause a lot of disk thrash as the heads have to constantly hop among all the different files being concurrently read (on top of those being written). That, then, is likely to become your bottleneck! (having benchmarked that very thing in the past). We can guess if OP has a CPU like that, then whatever is being used to handle multiple 100GB fies, is not a terabyte-class SSD. Commented Dec 13, 2020 at 4:38
  • 1
    Exactly what I needed. I've executed this in a 32-cores server with SSD disks. It was 283 files ~800MB each. Took a couple minutes and worked great. Commented Jul 4, 2022 at 11:59
4

Your bottlenecks are: how fast it can read the file, how fast it can compress it, and how fast it can write it or transfer it to the destination media, perhaps over the network.

First thing to do would be to run the gzip command while monitoring the output of

vmstat 1

in another terminal. You'll see if your CPU is maxed out, how many cores it uses, and how much MB/s it reads and writes. Also monitor vmstat while copying a huge file to get an idea of your hard drive's max read/write speed. Then you'll know if the operation is cpu bound or io bound.

You can also use

time gzip ...

It will tell you how much cpu time it used versus the total time, so that gives useful hints on whether it's cpu bound or waiting for IO.

If you intend to transfer the compressed file to another harddisk or over the network, then it makes sense to do so while compressing it, instead of using a separate copy operation. If the destination drive is local, just use the adequate gzip syntax ; if it is remote you can use a network share or:

gzip -c file.txt | ssh user@ip "cat > destfile.gz"

This will gzip the file and transfer it in one pipelined operation, which is faster than two separate steps.

Now, watch vmstat and determine if the operation is io-bound, network-bound, or cpu-bound. I recommend to install the utility "pv" and use it like this:

gzip -c file.txt | pv | ssh user@ip "cat > destfile.gz"

pv will display how many MB/s of compressed data is transferred through the network. You can test your HDD read, network, and write on the other end with this:

cat file.txt | pv | ssh user@ip "cat > destfile.gz"

You can test your HDD network and write on the other end with this:

cat /dev/zero | pv | ssh user@ip "cat > destfile.gz"

...and you can test only the network with this:

cat /dev/zero | pv | ssh user@ip "cat > /dev/null"

Now you should have a much better idea of what slows it down. Note if you use samba network shares you should also test the throughput:

cat /dev/zero | pv > /mnt/share/filename

...just in case your network share performance is clobbered by a misconfiguration, it's always nice to know.

If you determine that the problem is really gzip's speed, then the solution is to use a faster multithreaded compressor like zstandard. You can also use a faster compression setting, since saving a few GB of harddisk space is probably much less important than saving a few hours.

If compressed file size is less important than how long it takes, the optimum solution is a compression that is fast enough to saturate either the disk or network bottleneck.

For example if you have a slow network and a fast disk, and you have cpu to spare, using a higher compression setting will make it faster by transferring a smaller amount of compressed data. But if you have a fast network and a slow CPU, then a lower compression setting will use less CPU so it will be faster.

Now, where does this 100GB file comes from? This is not a common file size... and it hints that you should really be using rsync in delta mode.

3

Your storage is slowing you down.

gzip file.txt

Assuming file.txt is 100GB like you said, this requires reading 100GB from file.txt and writing somewhat less to file.txt.gz in the same directory.

If you have a spinning drive, that's a lot of head thrashing. If you have an SSD, it still only handles one read OR write at a time.

gzip -c file.txt > /mount/some_other_drive/file.txt.gz

The source drive can now deliver a steady stream of data, the destination drive can write the results without stopping every so many blocks to read the original file.

The destination must of course be on a different physical volume, and preferably a different interface to the computer.

4
  • I wouldn't be so quick to say it's the storage. Depending on compression settings, lower end CPUs could be a bottleneck. Commented Dec 12, 2020 at 7:11
  • 1
    @JohnLeuenhagen lower-end cpu could be the issue. Trying to read-write 150GB to the same directory will be slower than splitting the read and write between devices. There is no single bottleneck here, it's a + b + c.
    – peter
    Commented Dec 12, 2020 at 9:39
  • I estimated (in a comment) that the OP's CPU, using both cores (pigz), can probably average about 35MiB/s input compressing text. 35MiB/s sequential read + maybe 10 to 12MiB/s sequential write is likely something a rotational disk of similar vintage can handle, with the kernel doing readahead and write buffering to overlap actual DMA with pigz in user-space. So most read and write syscalls won't block, especially not with a single thread. Commented Dec 12, 2020 at 17:41
  • 1
    This would be useful if using a much faster compressor like zstdmt (multi-threaded zstdandard); rotational drives do much better with pure sequential read than with mixed read and write to separate locations. Commented Dec 12, 2020 at 17:42
1

You need to look at likely usage of the compressed file.

If its unlikely to need uncompressing, e.g. its a compressed version of a backup file, choose a compression method that is fast to compress but can be slow to decompress. gzip is not the only compression program.

If you expect to commonly uncompress the file, then is the work done to compress/decompress actually worth doing?

In addition, many large files do not compress particularly well - for example video, sound, graphics and some databases could be considered virtually incompressible. It is worth finding out how much compression actually 'saves'.

1

First make sure whether you are bottlenecked by the CPU of by I/O. If by I/O, the only way to improve, is to read from one and write to another drive, if available. Your CPU is quite weak, I see little point trying any modern compression algorithm, as those are generally more CPU-intensive. As your CPU has two cores, you can divide the file into two parts and gzip them in parallel. When done, you can concatenate these two gz files back into one (this is a neat feature of gzip format). Assumption here is that you have enough I/O. You may also play around with gzip compression levels down to 1.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .