1

Some GNU coreutils utilities like sort and shuf use a file as what effectively serves a seed. Does the size of the file matter?

The recommended way, https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html, uses an openssl-based method that takes a rather long time.

What if I just used a 6-letter word as below? Does this affect the ability of said utilities to create pseudo-randomness?

shuf -i1-10 --random-source=<(echo durian)

2

If you provide a fixed string as the random source, then it will "randomise" in the same way every single time. To prove this, let's test it.

$ printf '%s\n' a b c | shuf --random-source=<(echo durian)
b
c
a

On my system, the output is the same every time I run the command above. (I suspect it might be different depending on the implementation, but it should still be the same every time.) You are hard-coding the randomness, as per this XKCD:

XKCD

It's not really random; it's just producing the same output every time. The size of the fixed-string source is irrelevant. It's still fixed.

There's relevant info in the link you provide relating to the random quality of the random source:

/dev/urandom suffices for most practical uses, but applications requiring high-value or long-term protection of private data may require an alternate data source like /dev/random or /dev/arandom.

The latter two options are "more random" than the first. The implication is that the more random the source, the more random the shuffling. Hence, fixed strings are not particularly robust.

With shuf specifically, the length of the fixed string is relevant. For example, the following fails.

shuf -i1-19 --random-source=<(echo durian)

However, if you restrict output to -n16, it works, but -n17 fails. I tested a few different words and permutations, and as I reduce the number of characters in the source, the maximum -n goes down.

source length     max -n
7                 16
6                 13
5                 10
4                  8
3                  5
2                  3
1                  1
0                  0

I'm not sure of the direct relationship, but presumably additional sorted items (in -n) require more source characters as seeds. Nevertheless, in shuf at least, once you pass this minimum threshold, each additional character makes no difference to the randomness per se. In the above example, if you change the 50th character the output will still be the same.

  • Thanks Spearhawk for the detailed and interesting answer. I understand using a fixed string, like in my example, or even the openssl method suggested in the documentation, ensures each invocation is deterministic by providing a seed value. This is by design; making each invocation's result deterministic is useful in some modeling applications. My question, which I might not have explained well, is more concerned about whether the length of the string makes a difference for furnishing randomness in one invocation. For instance, I wouldn't want a shuffle of 1-100 to be more or less sorted. – flow2k Feb 1 at 7:08
  • @flow2k I'm not really sure I understand the question. As per my answer: "The size of the fixed-string source is irrelevant. It's still fixed." Does that answer it? In one invocation, length has no relevance. If it's one specific fixed string, then it will have a constant output whether the length is 1 character or a billion. – Sparhawk Feb 1 at 8:34
  • No, it does not answer it 😆. Hmm. Are you familiar with the use of a seed value? Per my understanding, this "fix-string source" serves as exactly that. – flow2k Feb 2 at 20:33
  • My question is about the length of said "fixed-string source". You say "length has no relevance". It seems it does have some effect; I tried just now shuf -i1-19 --random-source=<(echo durian) and it gave me error, apparently because durian is too short. But I don't know enough about the internals of shuf to understand why. – flow2k Feb 2 at 20:36
  • Ah I understand now. Edited. – Sparhawk Feb 4 at 4:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.