Read everything in a pipe's buffer without waiting

Question

Is there a way to read everything already in the buffer of a pipe, pipe them to another program, and exit immediately without waiting for any more input?

It has to be binary safe. It's ok to require installing additional programs.

For example, the following command should output a and not b, and raise SIGPIPE on the second echo (at least if the system doesn't have too much load):

{ echo a; sleep 2s; echo b;} | { sleep 1s; my_command;}

Answer 1

If you do one read() system call on a pipe, it will hang if there's nothing in the pipe, but otherwise would return straight away with what's in there.

Now, for what's in there, with Linux 4.4 on my amd64 multicore system at least (YMMV with other systems or other versions of Linux), if one or more processes are currently doing a write() (or other writing system call) at the other end, possibly of more than the capacity of the pipe (64KiB by default on current versions of Linux), the scheduler will go back and forth between those processes and the one reading from the pipe several times during that read() system call and read() can return much more than the pipe capacity.

The dd command is the CLI interface to the read system call.

dd bs=1G count=1

(note that not all dd implementations support those G suffixes though) does one read() of size 1GiB from stdin (followed by one write() on stdout)

With GNU dd, you can avoid the blocking on empty pipe with iflag=nonblock. That sets the pipe in non-blocking mode, but note that it leaves it non-blocking afterwards which may not be desirable. For instance:

(date; sleep 2; date) |
  (sleep 1
   dd bs=1G count=1 status=none iflag=nonblock
   wc -c)

give:

Mon 23 Jan 22:11:30 GMT 2017
wc: 'standard input': Resource temporarily unavailable
0

As the pipe becomes non-blocking for wc as well.

And as said earlier, with or without nonblock, you can end up reading more than can possibly fit in the pipe:

$ (cat /dev/zero & cat /dev/zero & cat /dev/zero) |
     (sleep 1; dd bs=1G count=1) | wc -c
0+1 records in
0+1 records out
545914880 bytes (546 MB, 521 MiB) copied, 0.48251 s, 1.1 GB/s
545914880

(and you'd get different numbers from one run to the next).

Another approach on systems that support it is to use the FIONREAD ioctl() to query how much data is in the pipe before doing the read.

perl -e '
  require "sys/ioctl.ph";
  ioctl(STDIN, &FIONREAD, $n) or die "ioctl: $!\n";
  $n = unpack "L", $n;
  if ($n) {
    sysread STDIN, $text, $n or die "read: $!\n";
    print $text
  }'

Note that because it's done in two steps, if another process is also reading from the pipe at the same time, it may still end-up blocking if that other process empties the pipe in between the ioctl() and the read().

Answer 2

You can use a simple C program to read from stdin, write to stdout and quit without actually waiting for the EOF, and use it as an intermediate in the piping chain.

Here is a simple example:

main.c

#include <stdio.h>
#include <unistd.h>

#define BUFFER_SIZE 1024

int main(int argc, char **argv){
    char buffer[BUFFER_SIZE];

    ssize_t read_size;

    //Read up to BUFFER_SIZE bytes of what's currently at the stdin
    read_size = read(STDIN_FILENO, buffer, BUFFER_SIZE);
    if(read_size > 0){
        write(STDOUT_FILENO, buffer, read_size);
    }

    return 0;
}

Compile this program using gcc -o test main.c.

Then if you run { echo a; sleep 2s; echo b; } | { sleep 1s; ./test; } | { ./mycommand; }, mycommand will only receive the a, since it was the only character in the stdin when ./test read from it.

To verify it, you can run { echo a; sleep 2s; echo b; } | { sleep 1s; ./test; } | cat, which will only show an a in the screen.