Reading chunks from a pipe and execute command for each chunk

Question

I have a command (A) that outputs a stream of data to STDOUT. The data stream is small and most of the time nothing is sent. I want to pipe that data to another command (B) that immediately executes a command (C) for each chunk of data read (and pipe in the data in the read chunk) when read stops blocking.

The size of the chunks is irrelevant and can be whatever makes sense. Essentially I want xargs but without a delimiter, instead it should execute a command whenever read starts blocking instead.

Specifically command A for me is "tail -f logfile" and command C is "hexdump -C". The reason why I can't directly pipe into "hexdump -C" is that hexdump waits until it has read a full 16 bytes before it prints a new row. So I want to execute the command for every chunk read instead.

Answer 1

Raw read won't block if the pipe isn't empty -- it will return however many bytes are available in the pipe even if you ask for more. You can make use of that and run hexdump -C on each successful return from read.

#define _BSD_SOURCE
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>

#define N (8*512)  //pipe size on my system as shown in `ulimit -a`
static char buf[N];

int main(){
  ssize_t nread;
  FILE* p;

  for(;;){
    do { nread = read(0, buf, N); }while (nread < 0 && errno == EINTR);
    if(nread == 0) return 0;    //EOF
    if(nread < 0) goto error; 
    p = popen("hexdump -C", "w"); if(!p) goto error;
    if(fwrite(buf, sizeof(char), nread, p) != nread) goto error;
    pclose(p);
  }

  return 0;
  error:
    perror(""); return 1;
}

You can save this, e.g., as shovel.c and then make shovel (or gcc shovel.c -o shovel) and use it in your pipe.

Or if you're not interested in this, here's a ruby snippet that should do the same thing:

tail -f file |
ruby -e 'PSIZE=8*512; 
      while(bytes = STDIN.readpartial(PSIZE));
          IO.popen("hexdump -C","w") {|p| p.syswrite(bytes) }
      end '

Answer 2

If you want something cheesy that just prints hex of your data as it appears you can use this perl script:

perl -e '$| = 1; $i = 1;
while(1){
 sysread(STDIN,$ch,1) or exit;
 printf "%02x%s",ord($ch),$i++%16==0?"\n":" ";
}'

And if you really want to read until a block and run hexdump:

perl -e '$| = 1; $i = 1; $rin = ""; vec($rin, fileno(STDIN), 1) = 1;
open(F,"|hexdump -C") or die;
while(1){
 $nfound = select($rout=$rin, undef, undef, 0);
 if($nfound==1){
  sysread(STDIN,$data,999) or exit;
  syswrite(F,$data) or die;
 }else{
  close(F);
  open(F,"|hexdump -C") or die;
  select($rout=$rin, undef, undef, undef);
 }
}'