Fastest and most efficient way to get number of records (lines) in a gzip-compressed file

Question

I am trying to do a record count on a 7.6 GB gzip file. I found few approaches using the zcat command.

$ zcat T.csv.gz | wc -l
423668947

This works but it takes too much time (more than 10 minutes to get the count). I tried a few more approaches like

$ sed -n '$=' T.csv.gz
28173811
$ perl -lne 'END { print $. }' < T.csv.gz
28173811
$ awk 'END {print NR}' T.csv.gz
28173811

All three of these commands are executing pretty fast but giving an incorrect count of 28173811.

How can I perform a record count in a minimal amount of time?

Answer 1

The sed, perl and awk commands that you mention may be correct, but they all read the compressed data and counts newline characters in that. These newline characters have nothing to do with the newline characters in the uncompressed data.

To count the number of lines in the uncompressed data, there is no way around uncompressing it. Your approach with zcat is the correct approach and since the data is so large, it will take time to uncompress it.

Most utilities that deals with gzip compression and decompression will most likely use the same shared library routines to do so. The only way to speed it up would be to find an implementation of the zlib routines that are somehow faster than the default ones, and rebuild e.g. zcat to use those.

Answer 2

Use unpigz.

Kusalananda's answer is correct, you will need to uncompress that entire file to scan its contents. /bin/gunzip does this as fast as it can, on a single core. Pigz is a parallel implementation of gzip that can use multiple cores.

Sadly, the decompression itself of normal gzip files cannot be parallelized, but pigz does offer an improved version of gunzip, unpigz, that does related work such as reading, writing, and checksumming in a separate thread. In some quick benchmarks, unpigz is almost twice as fast as gunzip on my core i5 machine.

Install pigz with your favourite package manager, and use unpigz instead of gunzip, or unpigz -c instead of zcat. So your command becomes:

$ unpigz -c T.csv.gz | wc -l

All this assumes the bottleneck is the CPU, not the disk, of course.

Answer 3

The problem with all the pipelines is that you are essentially doubling the work. No matter how fast the decompression is, the data still need to be shuttled to another process.

Perl has PerlIO::gzip which allows you to read gzipped streams directly. Therefore, it might offer an advantage even if its decompression speed may not match that of unpigz:

#!/usr/bin/env perl

use strict;
use warnings;

use autouse Carp => 'croak';
use PerlIO::gzip;

@ARGV or croak "Need filename\n";

open my $in, '<:gzip', $ARGV[0]
    or croak "Failed to open '$ARGV[0]': $!";

1 while <$in>;

print "$.\n";

close $in or croak "Failed to close '$ARGV[0]': $!";

I tried it with a 13 MB gzip compressed file (decompresses to 1.4 GB) on an old 2010 MacBook Pro with 16 GB RAM and an old ThinkPad T400 with 8 GB RAM with the file already in the cache. On the Mac, the Perl script was significantly faster than using pipelines (5 seconds vs 22 seconds), but on ArchLinux, it lost to unpigz:

$ time -p ./gzlc.pl spy.gz 
1154737
real 4.49
user 4.47
sys 0.01

versus

$ time -p unpigz -c spy.gz | wc -l
1154737
real 3.68
user 4.10
sys 1.46

and

$ time -p zcat spy.gz | wc -l
1154737
real 6.41
user 6.08
sys 0.86

Clearly, using unpigz -c file.gz | wc -l is the winner here both in terms of speed. And, that simple command line surely beats writing a program, however short.

Answer 4

Kusalananda's answer is mostly correct. To count lines you need to search for newlines. However it is theoretically possible to search for newlines without completely uncompressing the file.

gzip uses DEFLATE compression. DEFLATE is a combination of LZ77 and Huffman encoding. There may be a way to figure out just the Huffman symbol node for newline and ignore the rest. There almost certainly is a way to look for newlines encoded using L277, keep a byte count and ignore everything else.

So IMHO its theoretically possible to come up with a solution more efficient than unpigz or zgrep. That being said its certainly not practical (unless someone has already done it).

Answer 5

Can be done using zgrep with -c flag, and $ parameter.

In this case -c instruct the command to output number of matched lines and the regex $ matches end of line so it matches every line or the file.

zgrep -c $ T.csv.gz 

As commented by @StéphaneChazelas - zgrep is only a script around zcat and grep and it should provide similar performance to the original suggestion of zcat | wc -l

Answer 6

As you can see, most answers tries to optimize what it can: the number of the context switches and inter-process IO. The reason is, this is the only what you can optimize here easily.

Now the problem is that its resource need is nearly negligible to the resource need of the decompression. This is why the optimizations won't really make anything faster.

Where it could be really accelerated, it would be a modified un-gzip (i.e. decompression) algorithm, which leaves out the actual production of the decompressed data stream; rather it only calculates the number of the newlines in the decompressed stream from the compressed one. It would be hard, it would require the deep knowledge of the gzip's algorithm (some combination of the LZW and Huffman compression algorithms). It is quite probable, that the algorithm doesn't make it possible to significantly optimize the decompression time with the lightening, that we only need to know the newline counts. Even if it would be possible, essentially a new gzip decompression library should have been developed (it doesn't exists until know).

The realistic answer to your question is, that no, you can't make it significantly faster.

Maybe you could use some parallelized gzip decompression, if it exists. It could use multiple CPU cores for the decompression. If it doesn't exist, it could be relatively easily developed.

For the xz, there exists a parallel compressor (pxz).