Why does dd from /dev/random give different file sizes?

Question

I am running the following command on an ubuntu system:

dd if=/dev/random of=rand bs=1K count=2

However, every time I run it, I end up with a file of a different size. Why is this? How can I generate a file of a given size filled with random data?

Answer 1

You're observing a combination of the peculiar behavior of dd with the peculiar behavior of Linux's /dev/random. Both, by the way, are rarely the right tool for the job.

Linux's /dev/random returns data sparingly. It is based on the assumption that the entropy in the pseudorandom number generator is extinguished at a very fast rate. Since gathering new entropy is slow, /dev/random typically relinquishes only a few bytes at a time.

dd is an old, cranky program initially intended to operate on tape devices. When you tell it to read one block of 1kB, it attempts to read one block. If the read returns less than 1024 bytes, tough, that's all you get. So dd if=/dev/random bs=1K count=2 makes two read(2) calls. Since it's reading from /dev/random, the two read calls typically return only a few bytes, in varying number depending on the available entropy. See also When is dd suitable for copying data? (or, when are read() and write() partial)

Unless you're designing an OS installer or cloner, you should never use /dev/random under Linux, always /dev/urandom. The urandom man page is somewhat misleading; /dev/urandom is in fact suitable for cryptography, even to generate long-lived keys. The only restriction with /dev/urandom is that it must be supplied with sufficient entropy; Linux distributions normally save the entropy between reboots, so the only time you might not have enough entropy is on a fresh installation. Entropy does not wear off in practical terms. For more information, read Is a rand from /dev/urandom secure for a login key? and Feeding /dev/random entropy pool?.

Most uses of dd are better expressed with tools such as head or tail. If you want 2kB of random bytes, run

head -c 2k </dev/urandom >rand

With older Linux kernels, you could get away with

dd if=/dev/urandom of=rand bs=1k count=2

because /dev/urandom happily returned as many bytes as requested. But this is no longer true since kernel 3.16, it's now limited to 32MB.

In general, when you need to use dd to extract a fixed number of bytes and its input is not coming from a regular file or block device, you need to read byte by byte: dd bs=1 count=2048.

Answer 2

From man 4 random on a RHEL 5 box:

When read, the /dev/random device will only return random bytes within the estimated number of bits of noise in the entropy pool.

I get files of size 213 bytes on that machine. Back to man 4 random:

When read, /dev/urandom device will return as many bytes as are requested.

I get 2048 bytes from every invocation of dd if=/dev/urandom of=rand bs=1K count=2

I conclude that the difference is due to how much entropy your machine generates between invocations of dd if=/dev/random ...

Answer 3

Why does dd drop data? ... Gilles has posed this engaging question about dd:
When is dd suitable for copying data? (or, when are read() and write() partial)
Here is an excerpt from that question:

    *...it's not difficult to put dd at fault; for example try this code:**
        yes | dd of=out bs=1024k count=10
    and check the size of the out file (it's likely to be well under 10MB).

---

Aside from my comment (at the end of your question), something like this is iteresting to watch... It catches your bytes in file $trnd. I've semi-arbitrarily chosen bs=8

Move your mouse and watch it speed up.
With my computer idle (AFK and no Network activity), and after exhausting the entropy pool, it took 2 hours 12 minutes to collect only 1192 bytes, at which point I cancelled it.

Then, with me moving the mouse continuously, it took a relatively much shorter 1 minute 15 seconds to collect the same number of bytes.

This shows pretty clearly that collecting entropy is not CPU speed based, but rather it is random events based, and that my Ubuntu system uses the mouse as one of its significant random factors.

get=2048
trnd=/tmp/$USER.rnd; >"$trnd"
while (( $(wc -c <"$trnd") < $get )) ;do
    dd if=/dev/random bs=8 count=1 2>/dev/null >>"$trnd"
    echo -n "itt: $((i+=1))  ct: "; wc -c <"$trnd"
done
truncate -s $get "$trnd"
echo -e "\nfinal count: "; wc -c <"$trnd"

Answer 4

dd is designed for blocking - it is usually the best tool at your disposal for reading from variable sized inputs if you need it done immediately because dd will not buffer current reads into some future write() (unless you very explicitly configure it that way with larger obs than ibs), but will instead write() everything it reads as soon as it read()s it (and optionally processes it).

Here are some important definitions:

• ibs=expr
  • Specify the input block size, in bytes, by expr (default is 512).
• obs=expr
  • Specify the output block size, in bytes, by expr (default is 512).
• bs=expr
  • Set both input and output block sizes to expr bytes, superseding ibs= and obs=. If no conversion other than sync, noerror, and notrunc is specified, each input block shall be copied to the output as a single block without aggregating short blocks.

So you see, when ibs and obs are defined together as bs then ibs takes precedence - but otherwise, if you are specific, then either obs or cbs does.

Here is an example in which ibs is most important. You might do something like this if you wanted to track how soon the /dev/random pool filled...

dd "ibs=$size" conv=sync "count=$lmt" \ 
    if=/dev/random of="$somefile"

As long as if='s target is readable at all, that will always result in the same sized output file, because dd will synchronize blocks read-in on nulls. In other words, if dd read()s for an input-block of $((size=10)) $((count=5)) times and the read() file returns 2 bytes, then 8 bytes, then 12 bytes, then 2 bytes, then 4 bytes, dd will write to its outfile something like

 2 read bytes 8NULs \
 8 read bytes 2NULs \
10 read bytes 0NULs \
 4 read bytes 6NULs \
 4 read bytes 6NULs

...because dd, by default, does not delay. So if you need to track in-stream and delimit the writes of some other process, dd is the tool for you.

If you're just writing some amount of data to a regular file then, contrary to other statements made here, you can also use dd for this - and fairly easily - but you'll need more than one and a reliable blocking factor.

For example, if you did:

{   dd ibs="$size" obs="${size}x$block_factor" |
    dd bs="${size}x$blockfactor" "count=$lmt"
}  <infile >outfile

...the first dd would buffer as many ibs="$size" input blocks as were necessary to fill at least one obs="${size}x$block_factor" output block for every write() to the pipe between it and the second dd. This means that the second dd can limit the output reliably with count="$lmt" because all of the write()s the first makes will match its i/o blocksize - regardless of how many read()s the first dd must do to make it so.

And that's how you can use dd to reliably read pipes or other types of special files - with just a little bit of math.