I am running the following command on an ubuntu system:

dd if=/dev/random of=rand bs=1K count=2

However, every time I run it, I end up with a file of a different size. Why is this? How can I generate a file of a given size filled with random data?

  • 1
    /dev/random will block if there is not enough entropy available to generate the number of digits you want. it simply takes time to gather that amount of high quality psuedo random "randomness"... Either use /dev/urandom for a less random "random" value, or check your entropy pool (in a loop, and wait as need be)... – Peter.O Feb 28 '12 at 21:18
  • See also this question. – Keith Thompson Feb 29 '12 at 2:43
  • 3
    just add iflag=fullblock – frostschutz Mar 23 '15 at 0:34

You're observing a combination of the peculiar behavior of dd with the peculiar behavior of Linux's /dev/random. Both, by the way, are rarely the right tool for the job.

Linux's /dev/random returns data sparingly. It is based on the assumption that the entropy in the pseudorandom number generator is extinguished at a very fast rate. Since gathering new entropy is slow, /dev/random typically relinquishes only a few bytes at a time.

dd is an old, cranky program initially intended to operate on tape devices. When you tell it to read one block of 1kB, it attempts to read one block. If the read returns less than 1024 bytes, tough, that's all you get. So dd if=/dev/random bs=1K count=2 makes two read(2) calls. Since it's reading from /dev/random, the two read calls typically return only a few bytes, in varying number depending on the available entropy. See also When is dd suitable for copying data? (or, when are read() and write() partial)

Unless you're designing an OS installer or cloner, you should never use /dev/random under Linux, always /dev/urandom. The urandom man page is somewhat misleading; /dev/urandom is in fact suitable for cryptography, even to generate long-lived keys. The only restriction with /dev/urandom is that it must be supplied with sufficient entropy; Linux distributions normally save the entropy between reboots, so the only time you might not have enough entropy is on a fresh installation. Entropy does not wear off in practical terms. For more information, read Is a rand from /dev/urandom secure for a login key? and Feeding /dev/random entropy pool?.

Most uses of dd are better expressed with tools such as head or tail. If you want 2kB of random bytes, run

head -c 2k </dev/urandom >rand

With older Linux kernels, you could get away with

dd if=/dev/urandom of=rand bs=1k count=2

because /dev/urandom happily returned as many bytes as requested. But this is no longer true since kernel 3.16, it's now limited to 32MB.

In general, when you need to use dd to extract a fixed number of bytes and its input is not coming from a regular file or block device, you need to read byte by byte: dd bs=1 count=2048.

  • Thanks for the tip on using head instead of dd. That allows me to still use /dev/random if I want to. Although /dev/urandom would probably be sufficient as you mention, it is nice to know how to use /dev/random should the need arise. – Daniel Feb 29 '12 at 17:47
  • on kernels since 3.16 /dev/urandom returns 32m per read(). – mikeserv Mar 23 '15 at 23:02

From man 4 random on a RHEL 5 box:

When read, the /dev/random device will only return random bytes within the estimated number of bits of noise in the entropy pool.

I get files of size 213 bytes on that machine. Back to man 4 random:

When read, /dev/urandom device will return as many bytes as are requested.

I get 2048 bytes from every invocation of dd if=/dev/urandom of=rand bs=1K count=2

I conclude that the difference is due to how much entropy your machine generates between invocations of dd if=/dev/random ...

  • Yeah, practically, unless he's at a real crypto application, @Daniel should use /dev/urandom. But I'm puzzled as to why dd if=/dev/random bs=1K count=2 stops when the entropy pool is apparently drained. From the docs, it should block until there's more entropy, so the dd will write out the file slowly, instead of just dumping out the current pool and exiting. – cjc Feb 28 '12 at 21:17
  • I wondered about that too, but it's consistent across RHEL, Slackware 13.1 and a pretty current Arch. The RHEL was x86_64, the others were 32-bit. Unfortunately the dd docs are in GNU info format, so I haven't read them all. – Bruce Ediger Feb 29 '12 at 2:23
  • It's also consistent on Gentoo too. – Matthew Scharley Feb 29 '12 at 4:09
  • 4
    @cjc: It's because when you call read(fd, mybuf, 1024) on a blocking FD it returns as soon as the underlying device returns some data. If there are 1024 bytes there to be read, it returns that. If there are only 201 bytes, it will return 201. If there are 0 bytes available, it will block until at least one byte becomes available, then return it/them. – Warren Young Feb 29 '12 at 20:19
  • @WarrenYoung does reading from /dev/random drain its contents? I assume so. – Michael Martinez May 16 at 16:55

Why does dd drop data? ... Gilles has posed this engaging question about dd:
When is dd suitable for copying data? (or, when are read() and write() partial)
Here is an excerpt from that question:

    *...it's not difficult to put dd at fault; for example try this code:**
        yes | dd of=out bs=1024k count=10
    and check the size of the out file (it's likely to be well under 10MB).

Aside from my comment (at the end of your question), something like this is iteresting to watch... It catches your bytes in file $trnd. I've semi-arbitrarily chosen bs=8

Move your mouse and watch it speed up.
With my computer idle (AFK and no Network activity), and after exhausting the entropy pool, it took 2 hours 12 minutes to collect only 1192 bytes, at which point I cancelled it.

Then, with me moving the mouse continuously, it took a relatively much shorter 1 minute 15 seconds to collect the same number of bytes.

This shows pretty clearly that collecting entropy is not CPU speed based, but rather it is random events based, and that my Ubuntu system uses the mouse as one of its significant random factors.

trnd=/tmp/$USER.rnd; >"$trnd"
while (( $(wc -c <"$trnd") < $get )) ;do
    dd if=/dev/random bs=8 count=1 2>/dev/null >>"$trnd"
    echo -n "itt: $((i+=1))  ct: "; wc -c <"$trnd"
truncate -s $get "$trnd"
echo -e "\nfinal count: "; wc -c <"$trnd"

dd is designed for blocking - it is usually the best tool at your disposal for reading from variable sized inputs if you need it done immediately because dd will not buffer current reads into some future write() (unless you very explicitly configure it that way with larger obs than ibs), but will instead write() everything it reads as soon as it read()s it (and optionally processes it).

Here are some important definitions:

  • ibs=expr
    • Specify the input block size, in bytes, by expr (default is 512).
  • obs=expr
    • Specify the output block size, in bytes, by expr (default is 512).
  • bs=expr
    • Set both input and output block sizes to expr bytes, superseding ibs= and obs=. If no conversion other than sync, noerror, and notrunc is specified, each input block shall be copied to the output as a single block without aggregating short blocks.

So you see, when ibs and obs are defined together as bs then ibs takes precedence - but otherwise, if you are specific, then either obs or cbs does.

Here is an example in which ibs is most important. You might do something like this if you wanted to track how soon the /dev/random pool filled...

dd "ibs=$size" conv=sync "count=$lmt" \ 
    if=/dev/random of="$somefile"

As long as if='s target is readable at all, that will always result in the same sized output file, because dd will synchronize blocks read-in on nulls. In other words, if dd read()s for an input-block of $((size=10)) $((count=5)) times and the read() file returns 2 bytes, then 8 bytes, then 12 bytes, then 2 bytes, then 4 bytes, dd will write to its outfile something like

 2 read bytes 8NULs \
 8 read bytes 2NULs \
10 read bytes 0NULs \
 4 read bytes 6NULs \
 4 read bytes 6NULs

...because dd, by default, does not delay. So if you need to track in-stream and delimit the writes of some other process, dd is the tool for you.

If you're just writing some amount of data to a regular file then, contrary to other statements made here, you can also use dd for this - and fairly easily - but you'll need more than one and a reliable blocking factor.

For example, if you did:

{   dd ibs="$size" obs="${size}x$block_factor" |
    dd bs="${size}x$blockfactor" "count=$lmt"
}  <infile >outfile

...the first dd would buffer as many ibs="$size" input blocks as were necessary to fill at least one obs="${size}x$block_factor" output block for every write() to the pipe between it and the second dd. This means that the second dd can limit the output reliably with count="$lmt" because all of the write()s the first makes will match its i/o blocksize - regardless of how many read()s the first dd must do to make it so.

And that's how you can use dd to reliably read pipes or other types of special files - with just a little bit of math.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.