I have a file formatted like this:

train/t/temple/east_asia/00000025.jpg 94
train/t/temple/east_asia/00000865.jpg 94
train/s/swamp/00000560.jpg 92
train/s/swamp/00000935.jpg 92
train/m/mountain/00000428.jpg 68
train/m/mountain/00000126.jpg 68

The last number is the class number. I have 50 different classes, and each class has 1,000 lines. I would like to take a random sample of size N from each class, and store the result in another text file.

  • Are lines grouped by class ? – don_crissti Mar 20 '16 at 22:03
  • Yes, indeed that is correct – apples-oranges Mar 20 '16 at 22:06

Since your lines are grouped by class, you could (with gnu tools)split the file into pieces and use the --fiter option to pipe each piece to shuf to extract N random lines from it:

split --filter='shuf -n N' infile > outfile

Note that split defaults to 1000 lines - which is what you need in this particular case. If the requirements change you'll have to pass the number of lines via -l
e.g. to split into pieces of 200 lines and extract 30 random lines from each piece:

split -l 200 --filter='shuf -n 30' infile > outfile
| improve this answer | |
  • Thanks for the elegant solution :) I cant upvote you since I don't have enough points, sorry about that! – apples-oranges Mar 20 '16 at 23:28
  • That looks like a UUOC; am I missing something? – G-Man Says 'Reinstate Monica' Mar 21 '16 at 7:17
  • 1
    Note that GNU split invokes $SHELL instead of sh to parse that shuf -n 30 command line. So you can speed it up by doing SHELL=/bin/sh split... – Stéphane Chazelas Nov 17 '17 at 15:42

You should be able to do that with just one awk command invocation:

To select p=50 lines at random out of every n=1000 lines:

awk -v n=1000 -v p=50 '
  BEGIN {srand(); remaining = p}
  NR > n {remaining = p; NR = 1}
  rand()*(n + 1 - NR) < remaining {
    print; remaining--
  }' < your-file
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.