0

I have a directory with many files. From each of these files I want a random sample and copy to a new directory with file names same as from which random sample was drawn.

0
0
mkdir -p random_samples
find dir -type f -exec sh -c 'for n do sort -R "$n" | head >"random_samples/${n##*/}"; done' sh {} +

This will look for all regular files in or below dir and will run a short shell script on these:

for n do
    sort -R "$n" | head >"random_samples/${n##*/}"
done

This short shell script will loop over the given pathnames (which will be regular files found by find) and will run sort -R on each of them. This will shuffle the lines about, and the head will produce the ten first lines of each shuffle. The output will go to the random_samples directory with a filename that is the same as the original file. No check for filename collisions in random_samples is done.

Instead of sort -R, you may also use shuf from GNU coreutils.

The only downside is that the ordering of the lines in the random samples will be random, i.e. the lines in the random samples will not be ordered according to their original ordering in the file.

To enforce the same ordering on the random samples as in the original file, we may replace our short shell script with

for n do
    awk -v OFS="\t" "{ print NR, \$0 }" "$n" | sort -R | head | sort -n |
    cut -f 2 >"random_samples/${n##*/}"
done

This starts with appending each line of the file with its line number (and a tab), and then we shuffle the lines and pick the first 10 as before. Then, the picked lines are sorted numerically and the line numbers are removed before the sample is saved.

Allowing find to run this:

find dir -type f -exec sh -c '
    for n do
        awk -v OFS="\t" "{ print NR, \$0 }" "$n" | sort -R | head | sort -n |
        cut -f 2 >"random_samples/${n##*/}"
    done' sh {} +

To pick more or less than 10 lines, change head to head -n NUM where NUM is the desired number of lines to pick.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.