I have written a C++ application that reads files line by line. It reads one line, does processing, then reads the next line.

I would like to add support for gzipped files, and am looking to an alternative to using C++ decompression libraries. One solution is to decompress the entire file, then read it as usual.

However, I'm dealing with very large files and I prefer the quick start-up time zcat provides by immediately decompressing the first lines of the file. The problem is that, left unchecked, zcat would decompress too much of the file too quickly for my application to keep up and the output buffer from the system command would quickly fill with a decompressed 10GB file while my application is still processing the first few lines.

Is it possible to pause zcat, or tell it to only read one line at a time, only advancing when the user indicates?

closed as off-topic by Gilles, Cory Klein, jasonwryan, rahmu, slm Aug 6 '13 at 23:39

  • This question does not appear to be about Unix or Linux within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    You are doing it wrong. You need to read up on basic concepts of Linux/Unix/POSIX. The first concept you need is a pipe. – n.m. Aug 6 '13 at 18:32
  • @n.m. I understand how pipes work in Linux, and that they have a capability similar to what I am looking for here in that if I zcat | someCommand, that zcat will only provide what someCommand is looking for. Is there a someCommand you are aware of that would actually answer my question? – Cory Klein Aug 6 '13 at 18:35
  • 2
    Your application is someCommand. – n.m. Aug 6 '13 at 18:36
  • 2
    If you don't want the user to type zcat |, you can use popen(2) system call to create the same thing from within your application. – n.m. Aug 6 '13 at 18:41
  • 4
    Programming questions are off-topic; you can ask about programming on Stack Overflow. But really you're overcomplicating things, you're just looking for the popen function. – Gilles Aug 6 '13 at 21:44

Quoted below from the manual page of pipe(7).

Pipe capacity

A pipe has a limited capacity. If the pipe is full, then a write(2) will block or fail, depending on whether the O_NONBLOCK flag is set (see below). Different implementations have different limits for the pipe capacity. Applications should not rely on a particular capacity: an application should be designed so that a reading process consumes data as soon as it is available, so that a writing process does not remain blocked.

In Linux versions before 2.6.11, the capacity of a pipe was the same as the system page size (e.g., 4096 bytes on i386). Since Linux 2.6.11, the pipe capacity is 65536 bytes.

The pipe has a specific capacity, probably 64k for modern linux kernel. If that capacity is reached, any writes are blocked until it some room is available after a read call. So, generally, zcat your_file.gz | your_program would only have 64k uncompressed at any given time and so you shouldn't worry about memory usage.

  • The thing is, my program is the thing that is doing the zcat via a system call. So it doesn't actually use the | character. But I do wonder if this is still applicable to the popen syscall mentioned in the comments by n.m. – Cory Klein Aug 6 '13 at 21:39
  • 2
    @CoryKlein This is a property of the pipe feature, which is what the | character in the shell triggers. It equally applies to other methods of creating pipes such as popen. – Gilles Aug 6 '13 at 21:43

If your application is only setup to read files and not stdin, you can use process substitution. It will function in a similar manor as a pipe, but the shell will present it to your application as a file. This feature is available in most modern shells, but is not POSIX. Here is an example:

yourscript <(zcat filename)

You can use pv to rate limit the data you pipe to zcat.

pv -L 5k -q < test.gz | zcat
  • That's irrelevant, zcat <test.gz will do just as well (except for not being bound by the completely arbitrary rate of 5kB/s). – Gilles Aug 6 '13 at 21:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.