Best method to collect a random sample from a collection of files

Question

Suppose there is a directory holding 300 data files. I want to randomly select 200 of those files and move them into another directory. Is there a way to do that under Unix/Linux?

Answer 1

If your system has shuf, you can use this quite conveniently (even handling ugly file names):

shuf -zen200 source/* | xargs -0 mv -t dest

If you don't have shuf but have a sort that takes -R, this should work:

find source -type f -print0 | sort -Rz | cut -d $'\0' -f-200 | xargs -0 mv -t dest

Answer 2

files=(*)
for (( i=0; i<200; i++ )); do
    keys=("${!files[@]}")
    rnd=$(( RANDOM % ${#keys[@]} ))
    key=${keys[$rnd]}
    mv "${files[$key]}" "$otherdir"
    unset files[$key]
done

Answer 3

Put all filenames into an array named "files" in bash:

files=( * )

size of array:

echo ${#files[@]}

define 2/3 of them as sample size:

take=$((2*${#files[@]}/3)) 

for i in $(seq 1 $take)
do
    r=$((RANDOM%${#files[@]})) 
    echo ${files[r]}
done

This will select duplicates, and isn't tested with filenames with blanks and such.

The simplest way to avoid duplicates is, to iterate over all files, and pick each one with 2/3 chance, but this will not necessarily lead to 200 files.

This will remove a file if it was chosen from the list and fulfill your requirements:

#!/bin/bash
files=( * )
# define 2/3 of them as sample size:
take=$((2*${#files[@]}/3)) 

while (( i < $take ))
do
    r=$((RANDOM%${#files[@]})) 
    f=${files[r]}
    if [[ -n $f ]]
    then 
        i=$((i+1))    
        echo ${files[r]}
        unset files[r]    
    fi
done

Answer 4

If this needs to be statistically random, you shouldn't use RANDOM % ${#keys[@]}. Consider:

1. $RANDOM has 32768 unique values
2. The first selection is 1 out of 300 elements
3. 32768 = 109 * 300 + 68

Thus, when selecting the first item, there's a 110/32768~=0.33569% chance for each of the 68 first elements, and 109/32768~=0.33264% chance for each of the other 232 elements to be selected. Picking is repeated several times with different chances, but biased towards the first elements whenever 32768 % ${#keys[@]} -ne 0, so the error compounds.

This should be unbiased, and works with any filename:

while IFS= read -r -d '' -u 9
do
    mv -- "$REPLY" /target/dir
done 9< <(find /source/dir -mindepth 1 -print0 | shuf -n 200 -z)