6

I am trying to get a set number of random lines that satisfy a condition.

e.g. if my file was:

a    1    5
b    4    12
c    2    3
e    6    14
f    7    52
g    1    8

then I would like exactly two random lines where the difference between column 3 and column 2 is greater than 3 but less than 10 (e.g. lines starting with a, b, e, and g would qualify)

How would I approach this?

awk (if something and random) '{print $1,$2,$3}'

11

You can do this in awk but getting the random selection of lines will be complex and will require writing quite a bit of code. I would instead use awk to get the lines that match your criteria and then use the standard tool shuf to choose a random selection:

$ awk '$3-$2>3 && $3-$2 < 10' file | shuf -n2
g    1    8
a    1    5

If you run this a few times, you'll see you get a random selection of lines:

$ for i in {1..5}; do awk '$3-$2>3 && $3-$2 < 10' file | shuf -n2; echo "--";  done
g    1    8
e    6    14
--
g    1    8
e    6    14
--
b    4    12
g    1    8
--
b    4    12
e    6    14
--
e    6    14
b    4    12
--

The shuf tool is part of the GNU coreutils, so it should be installed by default on most any Linux system and easily available for most any *nix.

  • 2
    Selecting a single value is easy (condition && rand() < 1 / ++n { selection = $0 }), but I'm not sure how to adapt that to selecting N, so yeah, shuf is probably the way to go. – Kevin Jun 22 '17 at 20:39
  • @Kevin ah, clever trick with the / ++n! However, the problem is that i) you need to also use srand() to set a random seed else you always get the same output and ii) you can sometimes get no output at all for a small file. I tried extending it to print two lines, but I still can't get past ii): awk 'BEGIN{srand()}{if($3-$2>3 && $3-$2 <10 && rand() < 1 / ++n){a[k++]=$0}if(k>1){print a[0]"\n"a[1]; exit}}' file and, in any case, that's more effort than it's worth, as you said. – terdon Jun 22 '17 at 20:48
  • 1
    I believe awk -v count=2 'BEGIN { srand() } $3 - $2 > 3 && $3 - $2 < 10 && rand() < count / ++n { if (n <= count) { s[n] = $0 } else { s[int(rand()*count)] = $0 } } END { for (i = 1; i <= count; i++) print s[i] }' file should work... but I'm not a statistician. – Kevin Jun 22 '17 at 21:14
  • @Kevin looks like it. Nice one! May as well make it into an answer. If only to show why we tend to use existing utilities when possible :) – terdon Jun 22 '17 at 21:16
  • sure, done. I did notice a bug in the comment, fixed in my answer. – Kevin Jun 22 '17 at 21:32
4

If you want a pure awk answer that only iterates through the list once:

awk -v count=2 'BEGIN { srand() } $3 - $2 > 3 && $3 - $2 < 10 && rand() < count / ++n { if (n <= count) { s[n] = $0 } else { s[1+int(rand()*count)] = $0 } } END { for (i in s) print s[i] }' input.txt

Stored in a file for easier reading:

BEGIN { srand() }
$3 - $2 > 3 &&
$3 - $2 < 10 &&
rand() < count / ++n {
    if (n <= count) {
        s[n] = $0 
    } else { 
        s[1+int(rand()*count)] = $0 
    } 
} 
END { 
    for (i in s) print s[i] 
}

The algorithm is a slight variation on Knuth's algorithm R; I'm pretty sure the change doesn't alter the distribution but I'm not a statistician so I can't guarantee it.

Commented for those less familiar with awk:

# Before the first line is read...
BEGIN { 
    # ...seed the random number generator.
    srand() 
}

# For each line:
# if the difference between the second and third columns is between 3 and 10 (exclusive)...
$3 - $2 > 3 &&
$3 - $2 < 10 &&
# ... with a probability of (total rows to select) / (total matching rows so far) ...
rand() < count / ++n {
    # ... If we haven't reached the number of rows we need, just add it to our list
    if (n <= count) {
        s[n] = $0 
    } else {
        # otherwise, replace a random entry in our list with the current line.
        s[1+int(rand()*count)] = $0 
    } 
} 

# After all lines have been processed...
END { 
    # Print all lines in our list.
    for (i in s) print s[i] 
}
  • Could you please explain this witchcraft for those uninitiated in awk :) – SumNeuron Jun 22 '17 at 22:49
  • 1
    Added some explanation. – Kevin Jun 22 '17 at 23:19
2

Here's one way to do it in GNU awk (which supports custom sort routines):

#!/usr/bin/gawk -f

function mycmp(ia, va, ib, vb) {
  return rand() < 0.5 ? 0 : 1;
}

BEGIN {
  srand();
}

$3 - $2 > 3 && $3 - $2 < 10 {
  a[NR]=$0;
} 

END {
  asort(a, b, "mycmp");
  for (i = 1; i < 3; i++) print b[i];
}

Testing with the given data:

$ for i in {1..6}; do printf 'Try %d:\n' $i; ../randsel.awk file; sleep 2; done
Try 1:
g    1    8
e    6    14
Try 2:
a    1    5
b    4    12
Try 3:
b    4    12
a    1    5
Try 4:
e    6    14
a    1    5
Try 5:
b    4    12
a    1    5
Try 6:
e    6    14
b    4    12
0

Posting a perl solution, as I don't see any reason why it must be in awk (except for the OP's wish):

#!/usr/bin/perl

use strict;
use warnings;
my $N = 2;
my $k;
my @r;

while(<>) {
    my @line = split(/\s+/);
    if ($line[2] - $line[1] > 3 && $line[2] - $line[1] < 10) {
        if(++$k <= $N) {
            push @r, $_;
        } elsif(rand(1) <= ($N/$k)) {
            $r[rand(@r)] = $_;
        }
    }
}

print @r;

This is a classic example of reservoir sampling. The algorithm was copied from here and modified by me to meet OP's specific wishes.

When saved in file reservoir.pl you run it with ./reservoir.pl file1 file2 file3 or cat file1 file2 file3 | ./reservoir.pl.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.