0

I have dd from GNU coreutils 8.32.

When I run { echo a; sleep 1; echo b; } | dd bs=4 count=1 then I get

a
0+1 records in
0+1 records out
2 bytes copied, 2.0381e-05 s, 98.1 kB/s

dd terminates during the sleep even though the block size was not reached and there was no EOF. The output b\n is lost. This does not happen if I remove either sleep or count=1.

In man dd I couldn't find anything that describes this behavior.

  1. Why doesn't dd count=1 wait till bs is reached or an EOF is encountered?
  2. How can I force dd to wait?
2

1 Answer 1

Reset to default
2

This isn't caused by dd's behaviour but operating system specific behaviour. That said it is behaviour specified by posix.

The value returned may be less than nbyte if the number of bytes left in the file is less than nbyte, if the read() request was interrupted by a signal, or if the file is a pipe or FIFO or special file and has fewer than nbyte bytes immediately available for reading. For example, a read() from a file associated with a terminal may return one typed line of data.

When you set bs=4 you instruct dd to read 4 bytes at a time, but that only means it requests 4 bytes per read(). Of the OS returns less, dd won't go back and read() a second time unless...

There is the iflag=fullblock which instructs dd to perform multiple read() operations to read an entire block.

14
  • Thanks, that helped a bit. Even though the specification of dd does not talk about such details. That dd happens to use read() the way it is seems more like a implementation detail we shouldn't be concerned about.
    – Socowi
    Mar 15, 2021 at 17:52
  • Anyways, since fullblock is not specified by posix, do you have any posix conform alternatives? I thought about dd bs=1 count=X instead of dd bs=X count=1 iflag=fullblock. In my tests this worked (albeit being far less efficient). If I understood correctly, read() blocks until there is at least one byte of input, right?
    – Socowi
    Mar 15, 2021 at 17:56
  • @Socowi no it won't return 0 unless less it's a non-blocking read which it shouldn't be. I'm afraid I don't have a posix alternative. You might want to check if BSD and Busybox both support that flag. Sometimes there are non-posix flags which are very common anyway.
    – Philip Couling
    Mar 15, 2021 at 17:58
  • @Socowi, and Philip, it's all about how dd behaves, and how it's specified. Right there in the POSIX text, second sentence: "It shall read the input one block at a time, using the specified input block size; it shall then process the block of data actually returned, which could be smaller than the requested block size." It's not like e.g. tail -c, which just reads a particular amount of bytes. So, unless you want that block-based behaviour, you probably shouldn't use dd. head -c isn't in POSIX, but it's rather common, and definitely exists in GNU.
    – ilkkachu
    Mar 15, 2021 at 18:45
  • @ilkkachu yes I feel this is hiding something in plain sight. Many users believe that bs * count = total data transferred. If it doesn't, and the same behaviour can be observed when reading from regular files then this invalidates a lot of advice about dd's behaviour.
    – Philip Couling
    Mar 15, 2021 at 19:30

Not the answer you're looking for? Browse other questions tagged .