How to avoid duplicate random number generation?

Question

TL;DR:

A program draws the same random seed when started twice simultaneously. How did it happen?

Details

I'm running an MCMC statistical analysis, so I execute the program (phylobayes) twice to get 2 independent replicates.

For doing so, my shell script detaches each replicate run, like this (schematically):

pb -d "inputdata" "replicate1" &
pid1=$!

pb -d "inputdata" "replicate2" &
pid2=$!

wait "$pid1"
wait "$pid2"

(Then, this script was submitted to a computing cluster (Debian 10) using slurm sbatch).

But many of my runs (like 30%) were started with the same random seed! as shown from the log files.

Phylobayes uses C++ Random::initRandom() command. [EDIT: actually, as pointed out this function has a custom definition inside the package].

Can it technically occur that the same random seed is taken? Does it use /dev/random or /dev/urandom ?

If yes, I will just insert a sleep command between starts;

if not, I have to understand what dumb mistake I made, but I don't see what it can be...

Answer 1

Can it technically occur that the same random seed is taken?

If you look at the function you linked, you will see that it uses the number of microseconds as the seed (modulo 10^6):

void Random::InitRandom(int seed)   {

    if (seed == -1) {
        struct timeval tod;
        gettimeofday(&tod, NULL);
        seed = tod.tv_usec; // <== this line
    }
    Seed = seed;
    srand(seed);
    ...
}

So, yes, if two instances of your software run that function on the same microsecond (or perhaps with an exactly 1 second difference, etc), you will get the same seed. That is plausible because you launch one instance immediately after another.

This is further aggravated by the fact that gettimeofday() often has a granularity coarser than 1 μs, so runs that happen very close to each other in time may have the same tv_usec value even if they don't occur on the exact same microsecond.

I think the best way to fix this is to build your own version with a different seed initialization algorithm that suits the way you use the software. E.g. instead of using microseconds, you can just increase the seed number by 1 on each subsequent run, or use /dev/random.

Alternatively, if you have GNU coreutils, you can use the sleep 0.001 command to pause the second execution for 1 millisecond, that should introduce the delay that should be sufficient for different instances to have different seeds.