Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Suppose there is a directory holding 300 data files. I want to randomly select 200 of those files and move them into another directory. Is there a way to do that under Unix/Linux?

share|improve this question
    
R can probably do this in the twinkling of an eye with list.files()... –  sr_ May 10 '12 at 14:29
3  
I'd vaguely plug together shuf and head (or just use shuf -n, should've read the man page...) –  Ulrich Schwarz May 10 '12 at 14:41

4 Answers 4

If your system has shuf, you can use this quite conveniently (even handling ugly file names):

shuf -zen200 source/* | xargs -0 mv -t dest

If you don't have shuf but have a sort that takes -R, this should work:

find source -type f -print0 | sort -Rz | cut -d $'\0' -f-200 | xargs -0 mv -t dest
share|improve this answer
3  
Ah yes, because where else would one look for shuffling than in a tool for sorting. (At least shuf isn't called tros because it does the opposite of sorting.) –  Ulrich Schwarz May 10 '12 at 16:52
2  
There's no such thing as the opposite of sorting (in the same sense as there's no such thing as "no weather"). Random is still sorted, it's just sorted randomly. –  Plutor May 10 '12 at 20:43
files=(*)
for (( i=0; i<200; i++ )); do
    keys=("${!files[@]}")
    rnd=$(( RANDOM % ${#keys[@]} ))
    key=${keys[$rnd]}
    mv "${files[$key]}" "$otherdir"
    unset files[$key]
done
share|improve this answer

Put all filenames into an array named "files" in bash:

files=( * )

size of array:

echo ${#files[@]}

define 2/3 of them as sample size:

take=$((2*${#files[@]}/3)) 

for i in $(seq 1 $take)
do
    r=$((RANDOM%${#files[@]})) 
    echo ${files[r]}
done

This will select duplicates, and isn't tested with filenames with blanks and such.

The simplest way to avoid duplicates is, to iterate over all files, and pick each one with 2/3 chance, but this will not necessarily lead to 200 files.

This will remove a file if it was chosen from the list and fulfill your requirements:

#!/bin/bash
files=( * )
# define 2/3 of them as sample size:
take=$((2*${#files[@]}/3)) 

while (( i < $take ))
do
    r=$((RANDOM%${#files[@]})) 
    f=${files[r]}
    if [[ -n $f ]]
    then 
        i=$((i+1))    
        echo ${files[r]}
        unset files[r]    
    fi
done
share|improve this answer
    
You might select the same file more than once. –  glenn jackman May 10 '12 at 14:43
    
Very nice shell script. To get around your problem of not getting 200 files, you probably want to use Reservoir Sampling: en.wikipedia.org/wiki/Reservoir_sampling I'm going to be weak and not include a shell script example of this. –  Bruce Ediger May 10 '12 at 14:51
    
@glennjackman: I wrote so, yes. Needed some minutes to figure out, how to remove entries from the array. –  user unknown May 10 '12 at 14:53
    
Minor caveat: $RANDOM can only have values 0 through 32767, so this will not work properly if you have more than 32768 files. Also, fetching is biased towards the first files. –  l0b0 May 10 '12 at 15:02
    
@l0b0: Requirements where, to pick 200 from 300. If the files aren't in the current directory, but on a file server, it will not work too. Different requirements, different answer. –  user unknown May 10 '12 at 15:05

If this needs to be statistically random, you shouldn't use RANDOM % ${#keys[@]}. Consider:

  1. $RANDOM has 32768 unique values
  2. The first selection is 1 out of 300 elements
  3. 32768 = 109 * 300 + 68

Thus, when selecting the first item, there's a 110/32768~=0.33569% chance for each of the 68 first elements, and 109/32768~=0.33264% chance for each of the other 232 elements to be selected. Picking is repeated several times with different chances, but biased towards the first elements whenever 32768 % ${#keys[@]} -ne 0, so the error compounds.

This should be unbiased, and works with any filename:

while IFS= read -r -d '' -u 9
do
    mv -- "$REPLY" /target/dir
done 9< <(find /source/dir -mindepth 1 -print0 | shuf -n 200 -z)
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.