Why does dd from /dev/urandom stops early? [duplicate]

Question

On my current Linux system (Debian Jessie amd64), I'm getting different behavior for dd using /dev/urandom (/dev/random behavior is properly documented). If I naively want 1G of random data:

$ dd if=/dev/urandom of=random.raw bs=1G count=1
0+1 records in
0+1 records out
33554431 bytes (34 MB) copied, 2.2481 s, 14.9 MB/s
$ echo $?
0

In this case only 34MB of random data are stored, while if I use multiple reads:

$ dd if=/dev/urandom of=random.raw bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 70.4749 s, 14.9 MB/s

then I properly get my 1G of random data.

The documentation for /dev/urandom is rather elusive:

A read from the /dev/urandom device will not block waiting for more entropy. As a result, if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack on the algorithms used by the driver. Knowledge of how to do this is not available in the current unclassified literature, but it is theoretically possible that such an attack may exist. If this is a concern in your application, use /dev/random instead.

I guess the documentation implies there is some sort of maximum read size for urandom.

I'm also guessing that the size of the entropy pool is 34MB on my system, which would explain why the first read of 1G failed at about 34MB.

But my question is how do I know the size of my entropy pool? Or is dd stopped by another factor (some kind of timing issue associated with urandom?).

Answer 1

If you check Reading from /dev/urandom gives EOF after 33554431 bytes and follow the discussion, it points to another bug report where Ted Tso states...

...that commit 79a8468747c5 causes reads larger than 32MB results in a only 32MB to be returned by the read(2) system call. That is, it results in a short read. POSIX always allows for a short read(2), and any program MUST check for short reads.

The problem with dd is that POSIX requires the count=X parameter, to be based on reads, not on bytes. This can be changed with iflag=fullblock.

As per gnu dd manual:

Note if the input may return short reads as could be the case when reading from
a pipe for example, ‘iflag=fullblock’ will ensure that ‘count=’ corresponds to
complete input blocks rather than the traditional POSIX specified behavior of
counting input read operations.

so if you add iflag=fullblock:

dd if=/dev/urandom of=random.raw bs=1G count=1 iflag=fullblock
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 65.3591 s, 16.4 MB/s

This is actually confirmed by dd, if you omit iflag and increase the count to get 32 reads, i.e. 32 x 33554431 bytes = 1073741792 bytes which is roughly 1G (or 1.1GB as per dd man page section on multiplicative suffixes), it will output a short warning:

dd if=/dev/urandom of=random.raw bs=1G count=32
dd: warning: partial read (33554431 bytes); suggest iflag=fullblock
0+32 records in
0+32 records out
1073741792 bytes (1.1 GB) copied, 59.6676 s, 18.0 MB/s

Answer 2

I wasn't previously aware of this oddity with urandom, but you are running into "short reads" with dd. See:

When is dd suitable for copying data? (or, when are read() and write() partial)

What you are doing in the first example is asking the Kernel to fill in 1GB of memory with random data in a single call to read(). Since memory is allocated lazily, you're essentially asking the kernel to allocate 1GB of memory into your process on that first call. There might be any number of reasons they stop early. Also, if it did what you asked you might run out of ram depending on what system you're trying this on.

Oh, and I doubt it has anything to do with your entropy pool. urandom is useful exactly because it will keep returning data even when the entropy pool is exhausted.