28

Suppose there is a directory holding 300 data files. I want to randomly select 200 of those files and move them into another directory. Is there a way to do that under Unix/Linux?

2
  • R can probably do this in the twinkling of an eye with list.files()...
    – sr_
    Commented May 10, 2012 at 14:29
  • 4
    I'd vaguely plug together shuf and head (or just use shuf -n, should've read the man page...) Commented May 10, 2012 at 14:41

6 Answers 6

36

If your system has shuf, you can use this quite conveniently (even handling ugly file names):

shuf -zen200 source/* | xargs -0 mv -t dest

If you don't have shuf but have a sort that takes -R, this should work:

find source -type f -print0 | sort -Rz | cut -d $'\0' -f-200 | xargs -0 mv -t dest
4
  • 7
    Ah yes, because where else would one look for shuffling than in a tool for sorting. (At least shuf isn't called tros because it does the opposite of sorting.) Commented May 10, 2012 at 16:52
  • 2
    There's no such thing as the opposite of sorting (in the same sense as there's no such thing as "no weather"). Random is still sorted, it's just sorted randomly.
    – Plutor
    Commented May 10, 2012 at 20:43
  • 1
    What is the "-zen200"? That's not in any of the documentation for shuf, or anywhere on the Internet, but your example doesn't work without it. Quite mystical.
    – SigmaX
    Commented Feb 17, 2015 at 21:56
  • 2
    @SigmaX Indeed, quite zen, isn't it. Hint: it's 3 separate flags.
    – Kevin
    Commented Feb 17, 2015 at 22:05
4

If this needs to be statistically random, you shouldn't use RANDOM % ${#keys[@]}. Consider:

  1. $RANDOM has 32768 unique values
  2. The first selection is 1 out of 300 elements
  3. 32768 = 109 * 300 + 68

Thus, when selecting the first item, there's a 110/32768~=0.33569% chance for each of the 68 first elements, and 109/32768~=0.33264% chance for each of the other 232 elements to be selected. Picking is repeated several times with different chances, but biased towards the first elements whenever 32768 % ${#keys[@]} -ne 0, so the error compounds.

This should be unbiased, and works with any filename:

while IFS= read -r -d '' -u 9
do
    mv -- "$REPLY" /target/dir
done 9< <(find /source/dir -mindepth 1 -print0 | shuf -n 200 -z)
2
files=(*)
for (( i=0; i<200; i++ )); do
    keys=("${!files[@]}")
    rnd=$(( RANDOM % ${#keys[@]} ))
    key=${keys[$rnd]}
    mv "${files[$key]}" "$otherdir"
    unset files[$key]
done
2

Kevin's solution works great! Something else I've used a lot because it find it easier to remember off the top of my head is something like:

cp `ls | shuf -n 200` destination
1

Put all filenames into an array named "files" in bash:

files=( * )

size of array:

echo ${#files[@]}

define 2/3 of them as sample size:

take=$((2*${#files[@]}/3)) 

for i in $(seq 1 $take)
do
    r=$((RANDOM%${#files[@]})) 
    echo ${files[r]}
done

This will select duplicates, and isn't tested with filenames with blanks and such.

The simplest way to avoid duplicates is, to iterate over all files, and pick each one with 2/3 chance, but this will not necessarily lead to 200 files.

This will remove a file if it was chosen from the list and fulfill your requirements:

#!/bin/bash
files=( * )
# define 2/3 of them as sample size:
take=$((2*${#files[@]}/3)) 

while (( i < $take ))
do
    r=$((RANDOM%${#files[@]})) 
    f=${files[r]}
    if [[ -n $f ]]
    then 
        i=$((i+1))    
        echo ${files[r]}
        unset files[r]    
    fi
done
5
  • You might select the same file more than once. Commented May 10, 2012 at 14:43
  • Very nice shell script. To get around your problem of not getting 200 files, you probably want to use Reservoir Sampling: en.wikipedia.org/wiki/Reservoir_sampling I'm going to be weak and not include a shell script example of this.
    – user732
    Commented May 10, 2012 at 14:51
  • @glennjackman: I wrote so, yes. Needed some minutes to figure out, how to remove entries from the array. Commented May 10, 2012 at 14:53
  • Minor caveat: $RANDOM can only have values 0 through 32767, so this will not work properly if you have more than 32768 files. Also, fetching is biased towards the first files.
    – l0b0
    Commented May 10, 2012 at 15:02
  • @l0b0: Requirements where, to pick 200 from 300. If the files aren't in the current directory, but on a file server, it will not work too. Different requirements, different answer. Commented May 10, 2012 at 15:05
0

One liner in bash:

ls original_directory/|sort -R|head -number_of_files_to_move|while read file; do cp "new_directory/"$file test; done
1
  • Please elaborate; U&L is a knowledge base. Commented Feb 22, 2017 at 9:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .