From the spec:
- If the
expr operand is specified and no conversions other than
notrunc are requested, the data returned from each input block shall be written as a separate output block; if the
read() returns less than a full block and the
sync conversion is not specified, the resulting output block shall be the same size as the input block.
So this is probably what causes your confusion. Yes, because
dd is designed for blocking, by default partial
read()s will be mapped 1:1 to partial
write()s, or else
syncd out on tail padding NUL or space chars to
bs= size when
conv=sync is specified.
This means that
dd is safe to use for copying data (w/ no risk of corruption due to a partial read or write) in every case but one in which it is arbitrarily limited by a
count= argument, because otherwise
dd will happily
write() its output in identically sized blocks to those in which its input was
read() until it
read()s completely through it. And even this caveat is only true when
bs= is specified or
obs= is not specified, as the very next sentence in the spec states:
- If the
expr operand is not specified, or a conversion other than
notrunc is requested, the input shall be processed and collected into full-sized output blocks until the end of the input is reached.
obs= arguments this can't matter - because
obs are both the same size by default. However, you can get explicit about input buffering by specifying different sizes for either and not specifying
bs= (because it takes precedence).
For example, if you do:
IN| dd ibs=1| OUT
...then a POSIX
write() in chunks of 512 bytes by collecting every singly
read() byte into a single output block.
Otherwise, if you do...
IN| dd obs=1kx1k| OUT
read() at maximum 512 bytes at a time, but
write() every megabyte-sized output block (kernel allowing and excepting possibly the last - because that's EOF) in full by collecting input into full-sized output blocks.
Also from the spec, though:
- Copy only n input blocks.
count= maps to
i?bs= blocks, and so in order to handle an arbitrary limit on
count= portably you'll need two
dds. The most practical way to do it with two
dds is by piping the output of one into the input of another, which surely puts us in the realm of reading/writing a special file regardless of the original input type.
An IPC pipe means that when specifying
[io]bs= args that, to do so safely, you must keep such values within the system's defined
PIPE_BUF limit. POSIX states that the system kernel must only guarantee atomic
write()s within the limits of
PIPE_BUF as defined in
limits.h. POSIX guarantees that
PIPE_BUF be at least ...
- Maximum number of bytes that is guaranteed to be atomic when writing to a pipe.
- Value: 512
...(which also happens to be the default
dd i/o blocksize), but the actual value is usually at least 4k. On an up-to-date linux system it is, by default, 64k.
So when you setup your
dd processes you should do it on a block factor based on three values:
- bs = ( obs =
PIPE_BUF or lesser )
- n = total desired number of bytes read
- count = n / bs
yes | dd obs=1k | dd bs=1k count=10k of=/dev/null
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 0.1143 s, 91.7 MB/s
You have to synchronize i/o w/
dd to handle non-seekable inputs. In other words, make pipe-buffers explicit and they cease to be a problem. That's what
dd is for. The unknown quantity here is
yes's buffer size - but if you block that out to a known quantity with another
dd then a little informed multiplication can make
dd safe to use for copying data (w/ no risk of corruption due to a partial read or write) even when arbitrarily limiting input w/
count= w/ any arbitrary input type on any POSIX system and without missing a single byte.
Here's a snippet from the POSIX spec:
- Specify the input block size, in bytes, by
expr (default is 512).
- Specify the output block size, in bytes, by
expr (default is 512).
- Set both input and output block sizes to
expr bytes, superseding
obs=. If no conversion other than
notrunc is specified, each input block shall be copied to the output as a single block without aggregating short blocks.
You'll also find some of this explained better here.