Parallel pausing and resuming?

Question

I have a pretty strange thing I'm seeing as I run 8 GPG encryption jobs with GNU parallel:

The command I've run is this:

find . -type f -not -iname "*.gpg" | sort |parallel --gnu -j 8 --workdir "$PWD" '
    echo "Encrypting {}..." ; 
    gpg --encrypt --recipient "[email protected]" "{}"
'

Why do the jobs seem to start and stop and start and stop rather than simply occupying all of the CPU time?

Answer 1

GPG needs some random bytes to encrypt. If you run out of entropy in the pool, that will cause GPG to pause.

--quick-random will use low quality random numbers, making the encryption insecure (and therefore useless, so use it only to test whether this is the problem, not in production), but will not run out. If using --quick-random will not pause, then this is the reason for your problem.

Answer 2

As Ole Tange wrote, gpg needs random data from /dev/random, which can slow down quite quickly if there is not enough entropy.

A good solution to this problem is haveged. If necessary, it provides new entropy to the kernel (and therefore to /dev/random).

Answer 3

Ole Tange's hypothesis that Gpg is blocking on a read to /dev/random is a good one. You can confirm it by looking at one of the gpg processes while it's blocked and checking what it's blocked on:

lsof -p1234
strace -s9999 -tt -p1234

(where 1234 is the PID of the gpg process). If you see something like this

…
gpg     1234 naftuli   4r   CHR    1,8      0t0       0 /dev/random
…
read(4, …

then this is the problem.

Gpg has no option to use /dev/urandom instead of /dev/random. The difference between those two devices is that /dev/urandom never blocks (even in the rare circumstances when it should) whereas /dev/random often blocks (even in the common cases where it shouldn't). For the long story, read Is a rand from /dev/urandom secure for a login key?

A quick workaround would be to make a copy of the gpg binary, replace /dev/random by /tmp/random (or anything else with the same length, which unfortunately excludes /dev/urandom), and create a symbolic link /tmp/random -> /dev/urandom.

Answer 4

Swap, I/O?

What does 1 of the threads look like? My first instinct is that the system is waiting for some resource to become available, perhaps disks? swap?

Built-in delay?

The other thought is that gpg may have some delays built in for no other reason than to add to the expense of creating keys, and so the delays are causing your lull in CPU usage.

These would be a form of a NOOP, where the key generation algorithm waits idle for some period of time to pass before proceeding.

Also you can gain insight into what's going on by running 1 of the gpg processes using strace to see what system calls are being made.

Example

$ strace gpg --encrypt --recipient "[email protected]" "..."

Buffer?

The other thing I would be suspicious of is buffering. Perhaps there is a buffer in your pipeline that is being exhausted faster than can be replenished, so the gpg processes are being starved for work to do.

You could use a tool such as pv to ferret out this issue by putting it after the output from find.

Example

$ find .... | sort | pv | ...

I'd look at these switches:

   -a, --average-rate
          Turn the average rate counter on.  This will display the average 
          rate of data transfer so far.

   -b, --bytes
          Turn the total byte counter on.  This will display the total 
          amount of data transferred so far.