14

This question already has an answer here:

On my current Linux system (Debian Jessie amd64), I'm getting different behavior for dd using /dev/urandom (/dev/random behavior is properly documented). If I naively want 1G of random data:

$ dd if=/dev/urandom of=random.raw bs=1G count=1
0+1 records in
0+1 records out
33554431 bytes (34 MB) copied, 2.2481 s, 14.9 MB/s
$ echo $?
0

In this case only 34MB of random data are stored, while if I use multiple reads:

$ dd if=/dev/urandom of=random.raw bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 70.4749 s, 14.9 MB/s

then I properly get my 1G of random data.

The documentation for /dev/urandom is rather elusive:

A read from the /dev/urandom device will not block waiting for more entropy. As a result, if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack on the algorithms used by the driver. Knowledge of how to do this is not available in the current unclassified literature, but it is theoretically possible that such an attack may exist. If this is a concern in your application, use /dev/random instead.

I guess the documentation implies there is some sort of maximum read size for urandom.

I'm also guessing that the size of the entropy pool is 34MB on my system, which would explain why the first read of 1G failed at about 34MB.

But my question is how do I know the size of my entropy pool? Or is dd stopped by another factor (some kind of timing issue associated with urandom?).

marked as duplicate by Gilles linux Jan 13 '15 at 23:51

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Note that there are 1024 megabytes in a gigabyte, not 1000. You don't have 1 gigabyte of bytes – Mooing Duck Jan 13 '15 at 21:26
  • 4
    Could someone please tell the admin that /dev/random is not /dev/urandom. This question is not a duplicate IMHO. – malat Jan 14 '15 at 6:54
  • 1
    @Gilles - the answer you link to (your own) and which you closed this question in favor of states you can run dd, because /dev/urandom happily returns as many bytes as requested. Strange then, isn't it, that this question pointedly puts the lie to that statement, and so does another answer here? The correct thing to do is to close the question you like to, I think, in favor of this one - because here, at least, the answers provided are correct. – mikeserv Mar 24 '15 at 4:28
  • @malat The issue is with the use of dd, and it's the same in both cases: the device isn't returning as much data that dd requested, but dd ignores that. The issue arises with smaller sizes when you use random than when you use urandom, but it's the same problem (and the answers demonstrate that by giving the same explanation). – Gilles Mar 24 '15 at 8:58
  • 2
    @Gilles - actually, the answers here and there differ significantly in that one here notes: commit 79a8468747c5 causes reads larger than 32MB results in a only 32MB to be returned by the read(2) system call. whereas yours states /dev/urandom happily returns as many bytes as requested.. This is particularly relevant to the question - which is why only 32mb should be returned from a read of /dev/urandom. – mikeserv Mar 24 '15 at 15:19
18

If you check Reading from /dev/urandom gives EOF after 33554431 bytes and follow the discussion, it points to another bug report where Ted Tso states...

...that commit 79a8468747c5 causes reads larger than 32MB results in a only 32MB to be returned by the read(2) system call. That is, it results in a short read. POSIX always allows for a short read(2), and any program MUST check for short reads.

The problem with dd is that POSIX requires the count=X parameter, to be based on reads, not on bytes. This can be changed with iflag=fullblock.

As per gnu dd manual:

Note if the input may return short reads as could be the case when reading from
a pipe for example, ‘iflag=fullblock’ will ensure that ‘count=’ corresponds to
complete input blocks rather than the traditional POSIX specified behavior of
counting input read operations.

so if you add iflag=fullblock:

dd if=/dev/urandom of=random.raw bs=1G count=1 iflag=fullblock
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 65.3591 s, 16.4 MB/s

This is actually confirmed by dd, if you omit iflag and increase the count to get 32 reads, i.e. 32 x 33554431 bytes = 1073741792 bytes which is roughly 1G (or 1.1GB as per dd man page section on multiplicative suffixes), it will output a short warning:

dd if=/dev/urandom of=random.raw bs=1G count=32
dd: warning: partial read (33554431 bytes); suggest iflag=fullblock
0+32 records in
0+32 records out
1073741792 bytes (1.1 GB) copied, 59.6676 s, 18.0 MB/s
  • 1
    I know nothing of linux, but find myself curious at the 1.1 GB result for bs=1G count=1 iflag=fullblock – Mooing Duck Jan 13 '15 at 19:58
  • 1
    @MooingDuck - what's wrong with the 1.1 GB result ? As per dd man page: ...BYTES may be followed by the following multiplicative suffixes: ...GB = 1000*1000*1000, G = 1024*1024*1024, and so on... so 1073741792 bytes translates roughly to 1.1 GB or 1 G. – don_crissti Jan 13 '15 at 20:16
  • 1
    The op's results say 1048576000 bytes (1.0 GB), while yours says 1073741792 bytes (1.1 GB), and... oh. He only wrote 1000 1M chunks, instead of 1024. Nevermind, your code is right. – Mooing Duck Jan 13 '15 at 21:25
4

I wasn't previously aware of this oddity with urandom, but you are running into "short reads" with dd. See:

When is dd suitable for copying data? (or, when are read() and write() partial)

What you are doing in the first example is asking the Kernel to fill in 1GB of memory with random data in a single call to read(). Since memory is allocated lazily, you're essentially asking the kernel to allocate 1GB of memory into your process on that first call. There might be any number of reasons they stop early. Also, if it did what you asked you might run out of ram depending on what system you're trying this on.

Oh, and I doubt it has anything to do with your entropy pool. urandom is useful exactly because it will keep returning data even when the entropy pool is exhausted.

  • 1
    The link shows the right direction but this problem is not about allocatable RAM. – Hauke Laging Jan 13 '15 at 16:39
  • 1
    adding iflag=fullblock did work for me though – malat Jan 13 '15 at 16:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.