2

I have a input file from which I want to randomly choose a line. In a column in every line is a value, and I want only a number of lines up to a maximum count of this value to be choosen randomly, not more.

So for simplicity, I started with a test file that looks like this:

1 1
2 1
3 1
4 1
5 2
6 2
7 3
8 4
9 4
10 4
11 5
12 5
13 5
14 5
15 5
16 6
17 6
18 7
19 8
20 8

Now I want to choose randomly lines, but only a maximum of for example 1 line per value in column 2:

1 1
6 2
7 3
9 4
13 5
17 6
18 7
19 8

This is the code I wrote to choose lines up to a maximum count:

#!/bin/bash

for ((i>0;i<8;i++))
do
gawk -v i="${i}" 'BEGIN{n=1} {if (i==$2 && n<=1) {n++;print $0;} if (n>1){next}} ' input >> output
done

How can I extend this to pick random lines up to the maximum?

  • Run the input through shuf first, that gives you the lines in random order? Then use awk to pick lines until you have one of each desired value. Alternatively, for each value, have awk (or grep) print all lines with that value, then shuf -n 1 to pick one out of those. – frostschutz Apr 25 '18 at 11:31
  • Are the numbers in the second column always non-decreasing? – choroba Apr 25 '18 at 11:33
  • @frostschutz I guess your second solution my problem could could be useful for my purpose. Thanks for the input! But is the statistics of shufflethe same as a random pick with replacement? @choroba Yes they are. These numbers stand for classes in which the lines are sort in. In my file they are currently not sorted by class number, but this is no problem to do. – quadrupoltensor Apr 25 '18 at 14:27
2

With GNU sort, for 1 as the maximum:

sort -R file | sort -uk2,2

For arbitrary values as the maximum:

sort -R file | awk -v max=2 '++count[$2] <= max' | head -n 8

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.