I have written a C++ application that reads files line by line. It reads one line, does processing, then reads the next line.
I would like to add support for gzipped files, and am looking to an alternative to using C++ decompression libraries. One solution is to decompress the entire file, then read it as usual.
However, I'm dealing with very large files and I prefer the quick start-up time
zcat provides by immediately decompressing the first lines of the file. The problem is that, left unchecked,
zcat would decompress too much of the file too quickly for my application to keep up and the output buffer from the system command would quickly fill with a decompressed 10GB file while my application is still processing the first few lines.
Is it possible to pause
zcat, or tell it to only read one line at a time, only advancing when the user indicates?