0

Trying to figure out how to have a shell script only take fix-length blocks out of stdin. I would have thought something like this would have worked, but it does not:

#!/bin/bash
value=0
while [ "$value" != "-1" ]; do
  read -r -d '' -n 20 value
  if [ "$value" != "-1" ]; then
    dd conv=notrunc status=none of=/path/bigfile.bin bs=1M count=1 seek=$value
  fi
done

In a nutshell I am trying to copy specific blocks from a large file to the same file a remote location. The data sent from the sending script has a block location (20 bytes) followed by a 1 MiB of the data to be written to that location. It is finished when the location is -1.

I have a setup that works fine if running a compiled C program, but I'd like to avoid this and run with the native shell commands. The problem is, it doesn't seem that dd consumes any data, and all of it is handled by the read command.

Can stdin be consumed by multiple commands?

6
  • stdin could be consumed by many programs, but you'd need to coordinate that very well. If you go down that road, it might make sense to convert the 1M data to some ASCII representation, send it over and then convert it back. By the way, it might be easier to help you if you post the other side of the pipe. Jan 20 at 5:06
  • 1
    "I am trying to copy specific blocks from a large file to the same file a remote location" sounds very much like you're trying to reimplement rsync. What's the true goal - for example, is it to make the remote file the same as the local one by sending as little data as possible?
    – roaima
    Jan 20 at 9:39
  • @roaima I wish I could simply use rsync, but rsync (as far as I know) only works on entire files. If it sees a 1 TB file has changed, it will simply send the 1 TB file--not just the parts that have changed. My goal is to only send the parts that have changed. Since I don't expect a lot of change between synchronizations, this incremental method is preferable. I'm not sure such a setup is useful outside a narrow use case--hence the specific question.
    – A. Que
    Jan 23 at 0:47
  • @EduardoTrápani The sending side is a C program. While I'm not opposed to posting the code, it is a bit lengthy and I thought that would clutter the question.
    – A. Que
    Jan 23 at 0:48
  • @A.Que if that's what rsync did it would be pointless. Rsync across a remote connection only sends differences. If you're copying between what looks like two local paths then it won't help you. Provide details about how you get the file to the remote system (for example scp) and I'll give you a really efficient rsync command that avoids all your complexity
    – roaima
    Jan 23 at 12:56

2 Answers 2

1

You could read everything with dd.

This should work (read ... value becomes value=$(dd bs=20 count=1 status=none)):

#!/bin/bash
value=0
while [ "$value" != "-1" ]; do
  value=$(dd bs=20 count=1 status=none)
  if [ "$value" != "-1" ]; then
    dd conv=notrunc status=none of=/path/bigfile.bin bs=1M count=1 seek=$value
  fi
done
2
  • I implemented your solution, but it does the same thing. On further investigation I found the answer to my problem isn't related to reading from stdin. Rather, dd needs to be run with the flag iflag=fullblock. Otherwise it only read/writes 65516 bytes (2^16 - 20). So now I'm not sure what to do about the topic as the answer to the asked question and the solution are unrelated.
    – A. Que
    Jan 23 at 6:37
  • Well, this answer helped, and it is an alternative. You could answer your own question with the solution. That would make it available to other people with similar issues. Jan 23 at 7:18
0

The answer to the actual question is yes, you can read incrementally from stdin when piped.

The solution to the problem with the script was that the dd command wasn't consuming 1 MiB of data. As written dd would read 65516 bytes, which is 65536 (64 KiB) less the 20 bytes for the block size. After reading this limited quantity of data, dd would exit. The solution is using the flag iflag=fullblock. In addition, the check for -1 doesn't work as written. The full functional solution is:

value=0
while [ $value != -1 ]; do
  read -r -d '' -n 20 value
  value=$((value))
  if [ $value != -1 ]; then
    dd iflag=fullblock conv=notrunc status=none \
      of=/path/bigfile.bin bs=1M count=1 seek=$value
  fi
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.