Cfrtjryk

I am trying to get a set number of random lines that satisfy a condition.

e.g. if my file was:

a    1    5

b    4    12

c    2    3

e    6    14

f    7    52

g    1    8

then I would like exactly two random lines where the difference between column 3 and column 2 is greater than 3 but less than 10 (e.g. lines starting with a, b, e, and g would qualify)

How would I approach this?

awk (if something and random) '{print $1,$2,$3}'

You can do this in awk but getting the random selection of lines will be complex and will require writing quite a bit of code. I would instead use awk to get the lines that match your criteria and then use the standard tool shuf to choose a random selection:

$ awk '$3-$2>3 && $3-$2 < 10' file | shuf -n2

g    1    8

a    1    5

If you run this a few times, you'll see you get a random selection of lines:

$ for i in {1..5}; do awk '$3-$2>3 && $3-$2 < 10' file | shuf -n2; echo "--";  done

g    1    8

e    6    14

--

g    1    8

e    6    14

--

b    4    12

g    1    8

--

b    4    12

e    6    14

--

e    6    14

b    4    12

--

The shuf tool is part of the GNU coreutils, so it should be installed by default on most any Linux system and easily available for most any *nix.

If you want a pure awk answer that only iterates through the list once:

awk -v count=2 'BEGIN { srand() } $3 - $2 > 3 && $3 - $2 < 10 && rand() < count / ++n { if (n <= count) { s[n] = $0 } else { s[1+int(rand()*count)] = $0 } } END { for (i in s) print s[i] }' input.txt

Stored in a file for easier reading:

BEGIN { srand() }

$3 - $2 > 3 &&

$3 - $2 < 10 &&

rand() < count / ++n {

    if (n <= count) {

        s[n] = $0 

    } else { 

        s[1+int(rand()*count)] = $0 

    } 

} 

END { 

    for (i in s) print s[i] 

}

The algorithm is a slight variation on Knuth's algorithm R; I'm pretty sure the change doesn't alter the distribution but I'm not a statistician so I can't guarantee it.

Commented for those less familiar with awk:

# Before the first line is read...

BEGIN { 

    # ...seed the random number generator.

    srand() 

}



# For each line:

# if the difference between the second and third columns is between 3 and 10 (exclusive)...

$3 - $2 > 3 &&

$3 - $2 < 10 &&

# ... with a probability of (total rows to select) / (total matching rows so far) ...

rand() < count / ++n {

    # ... If we haven't reached the number of rows we need, just add it to our list

    if (n <= count) {

        s[n] = $0 

    } else {

        # otherwise, replace a random entry in our list with the current line.

        s[1+int(rand()*count)] = $0 

    } 

} 



# After all lines have been processed...

END { 

    # Print all lines in our list.

    for (i in s) print s[i] 

}

Here's one way to do it in GNU awk (which supports custom sort routines):

#!/usr/bin/gawk -f



function mycmp(ia, va, ib, vb) {

  return rand() < 0.5 ? 0 : 1;

}



BEGIN {

  srand();

}



$3 - $2 > 3 && $3 - $2 < 10 {

  a[NR]=$0;

} 



END {

  asort(a, b, "mycmp");

  for (i = 1; i < 3; i++) print b[i];

}

Testing with the given data:

$ for i in {1..6}; do printf 'Try %d:n' $i; ../randsel.awk file; sleep 2; done

Try 1:

g    1    8

e    6    14

Try 2:

a    1    5

b    4    12

Try 3:

b    4    12

a    1    5

Try 4:

e    6    14

a    1    5

Try 5:

b    4    12

a    1    5

Try 6:

e    6    14

b    4    12

Posting a perl solution, as I don't see any reason why it must be in awk (except for the OP's wish):

#!/usr/bin/perl



use strict;

use warnings;

my $N = 2;

my $k;

my @r;



while(<>) {

    my @line = split(/s+/);

    if ($line[2] - $line[1] > 3 && $line[2] - $line[1] < 10) {

        if(++$k <= $N) {

            push @r, $_;

        } elsif(rand(1) <= ($N/$k)) {

            $r[rand(@r)] = $_;

        }

    }

}



print @r;

This is a classic example of reservoir sampling. The algorithm was copied from here and modified by me to meet OP's specific wishes.

When saved in file reservoir.pl you run it with ./reservoir.pl file1 file2 file3 or cat file1 file2 file3 | ./reservoir.pl.