3

I have a command (A) that outputs a stream of data to STDOUT. The data stream is small and most of the time nothing is sent. I want to pipe that data to another command (B) that immediately executes a command (C) for each chunk of data read (and pipe in the data in the read chunk) when read stops blocking.

The size of the chunks is irrelevant and can be whatever makes sense. Essentially I want xargs but without a delimiter, instead it should execute a command whenever read starts blocking instead.

Specifically command A for me is "tail -f logfile" and command C is "hexdump -C". The reason why I can't directly pipe into "hexdump -C" is that hexdump waits until it has read a full 16 bytes before it prints a new row. So I want to execute the command for every chunk read instead.

1

2 Answers 2

3

Raw read won't block if the pipe isn't empty -- it will return however many bytes are available in the pipe even if you ask for more. You can make use of that and run hexdump -C on each successful return from read.

#define _BSD_SOURCE
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>

#define N (8*512)  //pipe size on my system as shown in `ulimit -a`
static char buf[N];

int main(){
  ssize_t nread;
  FILE* p;

  for(;;){
    do { nread = read(0, buf, N); }while (nread < 0 && errno == EINTR);
    if(nread == 0) return 0;    //EOF
    if(nread < 0) goto error; 
    p = popen("hexdump -C", "w"); if(!p) goto error;
    if(fwrite(buf, sizeof(char), nread, p) != nread) goto error;
    pclose(p);
  }

  return 0;
  error:
    perror(""); return 1;
}

You can save this, e.g., as shovel.c and then make shovel (or gcc shovel.c -o shovel) and use it in your pipe.

Or if you're not interested in this, here's a ruby snippet that should do the same thing:

tail -f file |
ruby -e 'PSIZE=8*512; 
      while(bytes = STDIN.readpartial(PSIZE));
          IO.popen("hexdump -C","w") {|p| p.syswrite(bytes) }
      end '
3
  • Yes, to clarify, the behaviour of read you describe the reason why I wanted to do the program you expressed with a command. So I was sort of looking for a command to do this and not a program because I feel like it's so trivial it should be possible to write a simple command to do it. Sorry if that wasn't clear enough. Commented Oct 7, 2015 at 17:40
  • @HannesLandeholm Got it, added a ruby snippet. I like playing with the lower levels. :) Commented Oct 7, 2015 at 17:50
  • I marked this as the answer although I'm a bit saddened that there's no native unix program that you can just chain with your pipes and you have to use a real programming language. Commented Oct 8, 2015 at 13:27
3

If you want something cheesy that just prints hex of your data as it appears you can use this perl script:

perl -e '$| = 1; $i = 1;
while(1){
 sysread(STDIN,$ch,1) or exit;
 printf "%02x%s",ord($ch),$i++%16==0?"\n":" ";
}'

And if you really want to read until a block and run hexdump:

perl -e '$| = 1; $i = 1; $rin = ""; vec($rin, fileno(STDIN), 1) = 1;
open(F,"|hexdump -C") or die;
while(1){
 $nfound = select($rout=$rin, undef, undef, 0);
 if($nfound==1){
  sysread(STDIN,$data,999) or exit;
  syswrite(F,$data) or die;
 }else{
  close(F);
  open(F,"|hexdump -C") or die;
  select($rout=$rin, undef, undef, undef);
 }
}'
2
  • Good to know, but I'd like to see an answer to the original question. It would be useful for more than just hexdumps.
    – Wildcard
    Commented Oct 7, 2015 at 17:16
  • @Wildcard I added a less cheesy solution, tried on </dev/random.
    – meuh
    Commented Oct 7, 2015 at 17:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .