How to determine which processes are using how much entropy from /dev/urandom

Question

Doing fuser -v /dev/urandom tells me which processes currently have /dev/urandom open, but only that. Is there any was to determine anything about how much entropy each one is using up over time? For instance, it might be that one process is using up about 1 bit of entropy every minute, while a different one is using up about 8 bits per second; I'd like some way of determining that.

Answer 1

The short answer is 0, because entropy is not consumed.

There is a common misconception that entropy is consumed — that each time you read a random bit, this removes some entropy from the random source. This is wrong. You do not “consume” entropy. Yes, the Linux documentation gets it wrong.

During the life cycle of a Linux system, there are two stages:

1. Initially, there is not enough entropy. /dev/random will block until it thinks it has amassed enough entropy; /dev/urandom happily provides low-entropy data.
2. After a while, enough entropy is present in the random generator pool. /dev/random assigns a bogus rate of “entropy leek” and blocks now and then; /dev/urandom happily provides crypto-quality random data.

FreeBSD gets it right: on FreeBSD, /dev/random (or /dev/urandom, which is the same thing) blocks if it doesn't have enough entropy, and once it does, it keeps spewing out random data. On Linux, neither /dev/random nor /dev/urandom is the useful thing.

In practice, use /dev/urandom, and make sure when you provision your system that the entropy pool is fed (from disk, network and mouse activity, from a hardware source, from an external machine, …).

While you could try to read how many bytes get read from /dev/urandom, this is completely pointless. Reading from /dev/urandom does not deplete the entropy pool. Each consumer uses up 0 bits of entropy per any unit of time you care to name.

Answer 2

While not automated, you could use a tool like strace to watch for reads from the file descriptor(s) related to urandom. Then see how much data is read over a specific time period to get the read rate.

Answer 3

There are a few ways you could approach the problem if you don't know (or don't suspect) which process may be depleting entropy_available on Linux.

As mentioned, you could use strace, which is great for getting useful insight into which processes you might want to look at.

You could use auditd to audit which processes open /dev/random or /dev/urandom, but that won't tell you how much data is read (to prevent logging issues). Here are some commands do list the rules and then add two watches

auditctl -l
auditctl -w /dev/random
auditctl -w /dev/urandom
auditctl -l

Now SSH into the box (or do something else you know should resulting in opening /dev/urandom or similar, such as dd).

ausearch -ts recent | aureport -f

In my case, I see something like the following:

[root@metrics-d02 vagrant]# ausearch -ts recent | aureport -f

File Report
===============================================
# date time file syscall success exe auid event
===============================================
1. 07/01/20 01:13:36 /dev/urandom 2 yes /usr/bin/dd 1000 6383
2. 07/01/20 01:16:43 /dev/urandom 2 yes /usr/sbin/sshd -1 6389
3. 07/01/20 01:16:43 /dev/urandom 2 yes /usr/sbin/sshd -1 6388
4. 07/01/20 01:16:43 /dev/urandom 2 yes /usr/sbin/sshd -1 6390
5. 07/01/20 01:16:44 /dev/urandom 2 yes /usr/sbin/sshd 1000 6408

Do disable those watches

auditctl -W /dev/random
auditctl -W /dev/urandom

Remember though that this only will capture data for system calls that aren't read/write etc. so if there is anything that already has it open, you won't see it being read.

However, I noticed (using Prometheus and node_exporter) that I was still seeing a sawtooth pattern whereby the VM (CentOS 7 with nothing to collect entropy) was reporting entropy_available raising up to nearly 200 and upon doing so would plummet back down to 0.

Does lsof (of fuser if you prefer) offer anything?

[root@metrics-d02 vagrant]# lsof /dev/random /dev/urandom
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
chronyd 2184 chrony    3r   CHR    1,9      0t0 5339 /dev/urandom
tuned   2525   root    5r   CHR    1,9      0t0 5339 /dev/urandom

Note though the character device major and minor numbers; testing another way... (I'm not sure if this would be useful, just thinking in terms of things like Docker, which is not running on this VM)

[root@metrics-d02 vagrant]# ls -l /dev/*random
crw-rw-rw-. 1 root root 1, 8 Dec 19 01:24 /dev/random
crw-rw-rw-. 1 root root 1, 9 Dec 19 01:24 /dev/urandom

[root@metrics-d02 vagrant]# lsof | grep '1,[89]'
chronyd    2184           chrony    3r      CHR                1,9      0t0       5339 /dev/urandom
tuned      2525             root    5r      CHR                1,9      0t0       5339 /dev/urandom
gmain      2525  2714       root    5r      CHR                1,9      0t0       5339 /dev/urandom
tuned      2525  2715       root    5r      CHR                1,9      0t0       5339 /dev/urandom
tuned      2525  2717       root    5r      CHR                1,9      0t0       5339 /dev/urandom
tuned      2525  2754       root    5r      CHR                1,9      0t0       5339 /dev/urandom

Okay, so we have two processes, chronyd and tuned. Let's use strace. lsof told us chrony had /dev/urandom open for reading using file-discriptor 3

[root@metrics-d02 vagrant]# strace -p 2184 -f
strace: Process 2184 attached
select(6, [1 2 5], NULL, NULL, {98, 516224}
.... (I'm waiting)

So chronyd is waiting for some activity, with a timeout of 98 seconds from when it started this system call.

While I wait, I should highlight that my activity on the system is likely to create increase the kernels estimation of available random bits. (entropy_available)... so sit back and just watch the Prometheus graph...

We can repeat with tuned too... (this time adding some timestamps and a grep filter just for file-descriptor 5 (the read etc. calls would have this as the first argument)

[root@metrics-d02 vagrant]# strace -p 2525 -f -tt -T 2>&1 | grep '(5,'

Red Hat have a blog that further discusses CSPRNG (Cryptographically Secure Psuedo Random Number Generator). It discusses some other ways that processes can get access to random numbers:

• getrandom() system call <-- recommended for RHEL7.4+, high quality without blocking after entropy pool has been initialised
• /dev/random <-- will easily block
• /dev/urandom <-- problem when used before pool has been initiased. Will 'never block'; should be what most applications should use.
• AT_RANDOM <-- sets 16 random bytes once at execve-time

While AT_RANDOM is not useful, it is present for every process, so just the act of starting a process should drain at least a little bit.

You'll realise now that what I've shown above using lsof is not enough, it doesn't reveal use of getrandom(). But since getrandom() is a system-call, we should be able to reveal use of it using auditctl

[root@metrics-d02 vagrant]# auditctl -a exit,always -F arch=b64 -S getrandom
[root@metrics-d02 vagrant]# auditctl -l
-a always,exit -F arch=b64 -S getrandom
[root@metrics-d02 vagrant]# tail -F -n0 /var/log/audit/audit.log 
... (now we wait)

I got bored and ssh'd into the box, and I saw a lot of interesting cool stuff, but no getrandom(), which shouldn't come as a surprise as we saw it using the /dev/urandom API earlier.

So, trying to account for the depressions in the graph, nothing is opening /dev/*random, and nothing that has it open is currently using it, and nothing seems to be calling getrandom()... Is there anything else that would consume data from the [pool behind /dev/random]? What about the kernel? Consider functionality such as Address Space Layout Randomisation (ASLR):

https://access.redhat.com/solutions/44460 [requires subscription]

[root@metrics-d02 vagrant]# cat /proc/sys/kernel/randomize_va_space 
2

'2' here means that in addition to randomizing where things like mmap and stack (etc.) get loaded it will also enable heap randomization. What happens if we turn that off

[root@metrics-d02 vagrant]# echo 0 > /proc/sys/kernel/randomize_va_space
[root@metrics-d02 vagrant]# cat /proc/sys/kernel/randomize_va_space 
0

(Answer: same thing... perhaps someone else can illustrate this further)

The kernel will also be where AT_RANDOM is set. Here's a simple example which you can use strace to observe that it doesn't call /dev/*random or getrandom()

[vagrant@metrics-d02 ~]$ cat at_random.c 
#include <stdio.h>
#include <stdint.h>
#include <sys/auxv.h>

#define AT_RANDOM_LEN 16

int main(int argc, char *argv[])
{
    uintptr_t at_random;
    int i;

    at_random = getauxval(AT_RANDOM);

    for (i=0; i<AT_RANDOM_LEN; i++) {
        printf("%02x", ((uint8_t *)at_random)[i]);
    }
    printf("\n");

    /* show that it's a one-time thing */

    for (i=0; i<AT_RANDOM_LEN; i++) {
        printf("%02x", ((uint8_t *)at_random)[i]);
    }
    printf("\n");
}
[vagrant@metrics-d02 ~]$ make at_random
cc     at_random.c   -o at_random
[vagrant@metrics-d02 ~]$ ./at_random 
255f8d5711b9aecf9b5724aa53bc968b
255f8d5711b9aecf9b5724aa53bc968b
[vagrant@metrics-d02 ~]$ ./at_random 
ef4b25faf9f435b3a879a17d0f5c1a62
ef4b25faf9f435b3a879a17d0f5c1a62

Hope that's useful.

Practically, I would first look to Java workloads first as that's where I've typically been bitten by this the most. See https://blogs.oracle.com/luzmestre/why-does-my-weblogic-server-takes-a-long-time-to-start for an example.