How to gzip 100 GB files faster with high compression

Question

We have 100+ GB files on a Linux machine, and while trying to perform gzip using below command, gzip is taking minimum 1-2 hours to complete:

gzip file.txt

Is there a way we can make gzip to run fast with the same level of compression happening when we use gzip?

---

CPU: Intel(R) Core(TM) i3-2350M CPU @2.30 GHz

Answer 1

If you are using gzip, you use mostly one processor core (well, some parts of the task, like reading and writing data are kernel tasks and kernel will use another core). Have a look at some multicore-capable gzip replacements, e.g. MiGz (https://github.com/linkedin/migz) or Pigz (https://zlib.net/pigz/, for some longer explanation see also e.g. https://medium.com/ngs-sh/pigz-a-faster-alternative-to-gzip-for-big-files-d5909e46d659).

Answer 2

We have 100+ GB files and while trying to perform gzip using below command , gzip is taking minimum 1-2 hours to get complete

With CPU (taken from a comment): Intel® Core™ i3-2350M @ 2.30GHz, which has:

Number of Cores: 2; Number of Threads: 4

---

Your CPU sounds like a bottleneck according to this low score (benchmark), also to note this is a laptop CPU, quite an old one. In this setup, I expect a classic HDD instead of some modern SSD too, together with possibly low RAM, etc.

---

The conclusion is possibly no, you cannot do software-wise anything to get the higher performance of gzip on your computer without getting a lower compression ratio, of course.

---

Default compression setting is -6 if I am not mistaken, you could hit for example -2:

gzip -2 file.txt

and compare the results yourself. See the manual page for more settings.

---

UPDATE on pigz

Today, 2021-Jun-03, I myself needed to compress a rather large file of size 256 GB (239 GiB), and I somewhat tested gzip, bzip2, xz, and I found all of these being unable to take full advantage of my CPU (i7-7700HQ) and being fast, which is our goal in this Q&A.

In the end, I downloaded pigz (man page) from its home page, and compiled it simply by running make, then as I did not prefer to directly put it onto my PATH, so I created a Bash alias to the binary.

It might be useful to note how to watch the (possibly long) progress:

Example #1 (reading a prepared disk image and writing gzip'ed file in the same directory):

file=disk.img; pv < "$file" | pigz -2 > "$file".gz

Example #2 (reading disk directly and writing gzip'ed file in the the current directory):

dev=/dev/nvme0n1; file=disk.img.gz; pv < "$dev" | pigz -9 > "$file"

---

Conclusion

I now recommend using pigz, the parallel implementation of gzip, on very large files.

Answer 3

Do you specifically need gzip, or would other compression algorithms be an option? zstandard and lzop are both significantly faster than gzip.

Answer 4

As the others have pointed out, gzip is single-threaded.

If you have multiple files to process, you could do that in parallel:

find -type f -not -name '*.gz' -print0 | xargs -tr0n 1 -P$(nproc) gzip

• find: scan the file system tree
• -type f: filter for regular files
• -not: invert the next test
  • -name '*.gz': any files that are already compressed
• print0: print the name to stdout, followed by a NUL byte
• | pipe into
• xargs: read elements from stdin, and pass them as arguments to another program
• -t: write the commands run to the console
• -r: do not execute any command if no elements are present
• -0: expect elements to be separated by NUL bytes
• -n 1: give one element to each invocation
• -P ...: run as many commands in parallel
  • $(...): run this command and substitute its output
    • nproc: get the number of processors
• gzip: the command to be run for each file

Note that gzip is still rather fast as an algorithm, so you are likely to be I/O bound in this, and at the same time the compression isn't that great. If you are free to choose another compression method, you can use xz instead, which compresses way better, but needs more CPU time for that.

In theory, xz can parallelize internally, but that gives slightly worse compression:

xz -T$(nproc) *.txt

Answer 5

Your bottlenecks are: how fast it can read the file, how fast it can compress it, and how fast it can write it or transfer it to the destination media, perhaps over the network.

First thing to do would be to run the gzip command while monitoring the output of

vmstat 1

in another terminal. You'll see if your CPU is maxed out, how many cores it uses, and how much MB/s it reads and writes. Also monitor vmstat while copying a huge file to get an idea of your hard drive's max read/write speed. Then you'll know if the operation is cpu bound or io bound.

You can also use

time gzip ...

It will tell you how much cpu time it used versus the total time, so that gives useful hints on whether it's cpu bound or waiting for IO.

If you intend to transfer the compressed file to another harddisk or over the network, then it makes sense to do so while compressing it, instead of using a separate copy operation. If the destination drive is local, just use the adequate gzip syntax ; if it is remote you can use a network share or:

gzip -c file.txt | ssh user@ip "cat > destfile.gz"

This will gzip the file and transfer it in one pipelined operation, which is faster than two separate steps.

Now, watch vmstat and determine if the operation is io-bound, network-bound, or cpu-bound. I recommend to install the utility "pv" and use it like this:

gzip -c file.txt | pv | ssh user@ip "cat > destfile.gz"

pv will display how many MB/s of compressed data is transferred through the network. You can test your HDD read, network, and write on the other end with this:

cat file.txt | pv | ssh user@ip "cat > destfile.gz"

You can test your HDD network and write on the other end with this:

cat /dev/zero | pv | ssh user@ip "cat > destfile.gz"

...and you can test only the network with this:

cat /dev/zero | pv | ssh user@ip "cat > /dev/null"

Now you should have a much better idea of what slows it down. Note if you use samba network shares you should also test the throughput:

cat /dev/zero | pv > /mnt/share/filename

...just in case your network share performance is clobbered by a misconfiguration, it's always nice to know.

If you determine that the problem is really gzip's speed, then the solution is to use a faster multithreaded compressor like zstandard. You can also use a faster compression setting, since saving a few GB of harddisk space is probably much less important than saving a few hours.

If compressed file size is less important than how long it takes, the optimum solution is a compression that is fast enough to saturate either the disk or network bottleneck.

For example if you have a slow network and a fast disk, and you have cpu to spare, using a higher compression setting will make it faster by transferring a smaller amount of compressed data. But if you have a fast network and a slow CPU, then a lower compression setting will use less CPU so it will be faster.

Now, where does this 100GB file comes from? This is not a common file size... and it hints that you should really be using rsync in delta mode.

Answer 6

Your storage is slowing you down.

gzip file.txt

Assuming file.txt is 100GB like you said, this requires reading 100GB from file.txt and writing somewhat less to file.txt.gz in the same directory.

If you have a spinning drive, that's a lot of head thrashing. If you have an SSD, it still only handles one read OR write at a time.

gzip -c file.txt > /mount/some_other_drive/file.txt.gz

The source drive can now deliver a steady stream of data, the destination drive can write the results without stopping every so many blocks to read the original file.

The destination must of course be on a different physical volume, and preferably a different interface to the computer.

Answer 7

You need to look at likely usage of the compressed file.

If its unlikely to need uncompressing, e.g. its a compressed version of a backup file, choose a compression method that is fast to compress but can be slow to decompress. gzip is not the only compression program.

If you expect to commonly uncompress the file, then is the work done to compress/decompress actually worth doing?

In addition, many large files do not compress particularly well - for example video, sound, graphics and some databases could be considered virtually incompressible. It is worth finding out how much compression actually 'saves'.

Answer 8

First make sure whether you are bottlenecked by the CPU of by I/O. If by I/O, the only way to improve, is to read from one and write to another drive, if available. Your CPU is quite weak, I see little point trying any modern compression algorithm, as those are generally more CPU-intensive. As your CPU has two cores, you can divide the file into two parts and gzip them in parallel. When done, you can concatenate these two gz files back into one (this is a neat feature of gzip format). Assumption here is that you have enough I/O. You may also play around with gzip compression levels down to 1.