Create random data with dd and get "partial read warning". Is the data after the warning now really random?

Question

I create a 1TB file with random data with dd if=/dev/urandom of=file bs=1M count=1000000. Now I check with kill -SIGUSR1 <PID> the progress and get the following:

691581+0 Datensätze ein
691580+0 Datensätze aus
725174190080 Bytes (725 GB) kopiert, 86256,9 s, 8,4 MB/s
800950+1 Datensätze ein
800950+0 Datensätze aus
839856947200 Bytes (840 GB) kopiert, 99429,5 s, 8,4 MB/s
dd: warning: partial read (809620 bytes); suggest iflag=fullblock
803432+1 Datensätze ein
803431+1 Datensätze aus
842459273876 Bytes (842 GB) kopiert, 99791,3 s, 8,4 MB/s

I can't interpret the warning. What does it say? Is my file really random after the warning or is there a problem? What does +0 or +1 in 800950+1 Datensätze ein and 800950+0 Datensätze aus mean? After the warning it is +1. Is it a errorcount?

Answer 1

Summary: dd is a cranky tool which is hard to use correctly. Don't use it, despite the numerous tutorials that tell you so. dd has a “unix street cred” vibe attached to it — but if you truly understand what you're doing, you'll know that you shouldn't be touching it with a 10-foot pole.

dd makes a single call to the read system call per block (defined by the value of bs). There is no guarantee that the read system call returns as much data as the specified buffer size. This tends to work for regular files and block devices, but not for pipes and some character devices. See When is dd suitable for copying data? (or, when are read() and write() partial) for more information. If the read system call returns less than one full block, then dd transfers a partial block. It still copies the specified number of blocks, so the total amount of transfered bytes is less than requested.

The warning about a “partial read” tells you exactly this: one of the reads was partial, so dd transfered an incomplete block. In the block counts, +1 means that one block was read partially; since the output count is +0, all blocks were written out as read.

This doesn't affect the randomness of the data: all the bytes that dd writes out are bytes that it read from /dev/urandom. But you got fewer bytes than expected.

Linux's /dev/urandom accommodates arbitrary large requests (source: extract_entropy_user in drivers/char/random.c), so dd is normally safe when reading from it. However, reading large amounts of data takes time. If the process receives a signal, the read system call returns before filling its output buffer. This is normal behavior, and applications are supposed to call read in a loop; dd doesn't do this, for historical reasons (dd's origins are murky, but it seems to have started out as a tool to access tapes, which have peculiar requirements, and was never adapted to be a general-purpose tool). When you check the progress, this sends the dd process a signal which interrupts the read. You have a choice between knowing how many bytes dd will copy in total (make sure not to interrupt it — no progress check, no suspension), or knowing how many bytes dd has copied so far, in which case you can't know how many more bytes it will copy.

The version of dd in GNU coreutils (as found on non-embedded Linux and on Cygwin) has a flag fullblock which tells dd to call read in a loop (and ditto for write) and thus always transfer full blocks. The error message suggests that you use it; you should always use it (in both input and output flags), except in very special circumstances (mostly when accessing tapes) — if you use dd at all, that is: there are usually better solutions (see below).

dd if=/dev/urandom iflag=fullblock of=file bs=1M count=1000000

Another possible way to be sure of what dd will do is to pass a block size of 1. Then you can tell how many bytes were copied from the block count, though I'm not sure what will happen if a read is interrupted before reading the first byte (which is not very likely in practice but can happen). However, even if it works, this is very slow.

The general advice on using dd is do not use dd. Although dd is often advertised as a low-level command to access devices, it is in fact no such thing: all the magic happens in the device file (the /dev/…) part, dd is just an ordinary tool with a high potential for misuse resulting in data loss. In most cases, there is a simpler and safer way to do what you want, at least on Linux.

For example, to read a certain number of bytes at the beginning of a file, just call head:

head -c 1000000m </dev/urandom >file

I made a quick benchmark on my machine and did not observe any performance difference between dd with a large block size and head.

If you need to skip some bytes at the beginning, pipe tail into head:

dd if=input of=output count=C bs=B seek=S
<input tail -c +$((S*B+1)) | head -c $((C*B)) >output

If you want to see progress, call lsof to see the file offset. This only works on a regular file (the output file on your example), not on a character device.

lsof -a -p 1234 -d 1
cat /proc/1234/fdinfo/1

You can call pv to get a progress report (better than dd's), at the expense of an additional item in the pipeline (performance-wise, it's barely perceptible).

Answer 2

The warning occurs when dd could not get enough data to fill a block in a single read. This happens with erratic or slow data sources, or sources that write data in smaller units than your requested blocksize.

There is no problem with data integrity, but the problem is that dd counts a partial read still as a read block.

If you are not using the count option, the warning hardly matters, it's just a performance consideration. But with count, you won't get the amount of data you requested. Due to partial reads, of will be smaller than count*bs at the end.

So when you use count, technically you should always use iflag=fullblock as well.

The +x should be the number of partial blocks.

Answer 3

< /dev/urandom \
dd ibs=4k obs=64k |
dd bs=64k count=16000000 >file

^That will just work. The misinformation otherwise had here is manifestly false. dd's buffers are explicit and so, to buffer input to count occurrences you need to explicitly buffer. That is all. Don't buy the fud.