Incremental reading from pipe

Question

Trying to figure out how to have a shell script only take fix-length blocks out of stdin. I would have thought something like this would have worked, but it does not:

#!/bin/bash
value=0
while [ "$value" != "-1" ]; do
  read -r -d '' -n 20 value
  if [ "$value" != "-1" ]; then
    dd conv=notrunc status=none of=/path/bigfile.bin bs=1M count=1 seek=$value
  fi
done

In a nutshell I am trying to copy specific blocks from a large file to the same file a remote location. The data sent from the sending script has a block location (20 bytes) followed by a 1 MiB of the data to be written to that location. It is finished when the location is -1.

I have a setup that works fine if running a compiled C program, but I'd like to avoid this and run with the native shell commands. The problem is, it doesn't seem that dd consumes any data, and all of it is handled by the read command.

Can stdin be consumed by multiple commands?

Answer 1

You could read everything with dd.

This should work (read ... value becomes value=$(dd bs=20 count=1 status=none)):

#!/bin/bash
value=0
while [ "$value" != "-1" ]; do
  value=$(dd bs=20 count=1 status=none)
  if [ "$value" != "-1" ]; then
    dd conv=notrunc status=none of=/path/bigfile.bin bs=1M count=1 seek=$value
  fi
done

Answer 2

The answer to the actual question is yes, you can read incrementally from stdin when piped.

The solution to the problem with the script was that the dd command wasn't consuming 1 MiB of data. As written dd would read 65516 bytes, which is 65536 (64 KiB) less the 20 bytes for the block size. After reading this limited quantity of data, dd would exit. The solution is using the flag iflag=fullblock. In addition, the check for -1 doesn't work as written. The full functional solution is:

value=0
while [ $value != -1 ]; do
  read -r -d '' -n 20 value
  value=$((value))
  if [ $value != -1 ]; then
    dd iflag=fullblock conv=notrunc status=none \
      of=/path/bigfile.bin bs=1M count=1 seek=$value
  fi
done