4

With the glob qualifiers in zsh one can sort the results of a filename globbing pattern match in various ways. For example, the pattern *(om) would match all non-hidden names in the current directory, ordered by modification timestamp.

However, I have at times wanted to have a way of having a randomised ordering (for example, to get a random sampling of files). As far as I have seen, there is no qualifier that does this directly.

Question: How may I get a randomised list of pathnames back from a zsh filename globbing pattern?

  • The word "sampling" is throwing me off here; do you want all the files, just in a random order, or do you want a subset (partial or complete) of the files? – Jeff Schaller Nov 8 at 16:12
  • @JeffSchaller With [1,50] (for example) in the qualifier, the first 50 matches would be retained. So *(om[1,50]) would return the 50 most recently modified files. To get a sample of random files, I would use something similar, but with om replaced by something that orders the matches in a random fashion. Or that's my hope anyway. – Kusalananda Nov 8 at 16:15
5

Use a random sort key (glob qualifier oe)::

*(Noe\''REPLY=$RANDOM,$RANDOM'\')

Explanation:

  • oe is followed by a one-character delimiter, a chunk of code, and another delimiter. The chunk of code may not contain the delimiter. Special characters need to be escaped so that they are not parsed while parsing the glob qualifiers themselves.
  • I use ' as the delimiter character (with a backslash because it needs to be escaped), and I wrap the code with ' to protect special characters that may be present. This way I can write arbitrary code as long as it doesn't contain '.
  • This chunk of code is executed for each matching file name in turn.
  • REPLY is initially set to the file name and whatever the code sets REPLY to is used as a sort key).

To sample $n elements randomly, add the […] qualifier:

*(Noe\''REPLY=$RANDOM,$RANDOM'\'[1,$n])

Occasionally some elements will get the same sort key, so all permutations are not equally likely, with a slight preference for keeping whatever results from applying the sort function to a list in directory order¹, but the bias is small. I use $RANDOM,$RANDOM as the sort key rather than $RANDOM to reduce the bias: $RANDOM is a 15-bit number and the bias would be noticeable as the number of files approaches 2^15.

Note that $RANDOM is good enough for sampling if the slight bias isn't a concern. It isn't suitable for anything that involves security. If you want a secure random permutation, use GNU coreutils's shuf. (If your favorite OS lacks a native shuf and you don't want to install GNU coreutils for some reason, you can try ibara's reimplementation instead.)

securely_permuted=("${(0)$(printf '%s\0' *(N) | shuf -z))}")

or a simpler version that may run into a command line length limit:

securely_permuted=("${(0)$(shuf -z -- *(N)))}")

¹ Experimentally the sort is stable (e.g. *(omoe\''REPLY=1'\') is equivalent to *(om), but the order from just *(oe\''REPLY=1'\') doesn't match *(oN). In any case, it's a small bias in favor of some particular order.

  • Nicely! Could you possibly say something about the code that gets executed for each match, the REPLY=$RANDOM,$RANDOM bit? It (the new value in $REPLY) is used as the sorting key? BTW, no, I'm not too concerned about high quality random numbers. – Kusalananda Nov 8 at 16:29
  • Re-reading your text and the manual, yes, it's setting a new sorting key. Thanks, much obliged. That wasn't too complicated. – Kusalananda Nov 8 at 16:32
  • 1
    @Kusalananda I think REPLY=… is relatively clear in the manual, but I added an explanation anyway, and I also explain the \''…'\' bit which the manual isn't helpful about. – Gilles 'SO- stop being evil' Nov 8 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.