4

I was trying to assign below command(which choose the random line from file) to a variable, but not working.

givinv@87-109:~$ head -$((${RANDOM} % `wc -l < file` + 1)) file | tail -1
cower
givinv@87-109:~$

Below the the error I am getting while trying to assign it to a variable.

givinv@87-109:~$ VARIA=`head -$((${RANDOM} % `wc -l < file` + 1)) file | tail -1`
bash: command substitution: line 1: unexpected EOF while looking for matching `)'
bash: command substitution: line 2: syntax error: unexpected end of file
bash: command substitution: line 1: syntax error near unexpected token `)'
bash: command substitution: line 1: ` + 1)) file | tail -1'
-l: command not found
givinv@87-109:~$

I even tried same with for loop and not working::

givinv@87-109:~$ for i in `head -$((${RANDOM} % `wc -l < file` + 1)) file | tail -1`;do echo $i ;done
bash: syntax error near unexpected token `<'
givinv@87-109:~$ 
  • 4
    You can use the command rl, if you don't wanna reinvent the wheel. – Ipor Sircer Nov 28 '16 at 10:40
  • 4
    @IporSircer, or GNU shuf – Stéphane Chazelas Nov 28 '16 at 11:22
  • 3
    Note that $RANDOM's maximum value in bash is 32767, so you would not be able to reach lines greater than that. Also note that it would not give you a uniform random distribution unless the number of lines in the file is a divisor of 32768 (is a power of 2). – Stéphane Chazelas Nov 28 '16 at 11:26
15

It's not working because you are attempting to nest unescaped backticks:

VARIA=`head -$((${RANDOM} % `wc -l < file` + 1)) file | tail -1`

That actually attempts to run head -$((${RANDOM} % as a single command first, and that gives you the 2 first errors:

$ VARIA=`head -$((${RANDOM} % `
bash: command substitution: line 1: unexpected EOF while looking for matching `)'
bash: command substitution: line 2: syntax error: unexpected end of file

Then, it tries to run

wc -l < file` + 1)) file | tail -1`

Which means it tries to evaluate + 1)) file | tail -1 (which is between the backticks), and that gives you the next errors:

$ wc -l < file` + 1)) file | tail -1`
bash: command substitution: line 1: syntax error near unexpected token `)'
bash: command substitution: line 1: ` + 1)) file | tail -1'

You can get around this by escaping the backticks:

VARIA=`head -$((${RANDOM} % \`wc -l < file\` + 1)) file | tail -1`

However, as a general rule, it is usually better not to use backticks at all. You should almost always use $() instead. It is more robust and can be nested indefinitely with a simpler syntax:

VARIA=$(head -$((${RANDOM} % $(wc -l < file) + 1)) file | tail -1)
2

just use this command

VARIA=$(head -n "$((${RANDOM} % $(wc -l < test) + 1))" test | tail -n 1)

to assign the result of a command to a variable we use $(...) (the ancient `...` form is harder to nest).

2

As another option for reading a random line from a file (and assigning it to a variable), consider a simplified reservoir sampling method, converted from thrig's perl implementation to awk, with Peter.O's seeding improvement:

VARIA=$(awk -v seed=$RANDOM 'BEGIN { srand(seed) } { if (rand() * FNR < 1) { line=$0 } } END { print line }' /usr/share/dict/words)

Here's the awk script, wrapped nicely:

awk -v seed=$RANDOM '
BEGIN { 
  srand(seed) 
}
{ 
  if (rand() * FNR < 1) { 
    line=$0
  } 
}
END { 
  print line 
}' /usr/share/dict/words

Because of the way awk's srand() works, you would get the same value if you run this script within the same second unless you seed it with something else random; here I've passed in bash's $RANDOM as the seed. Here I'm selecting words from /usr/share/dict/words, just as a source of text.

This method does not care how many lines are in the file (my local copy has 479,828 lines), so it should be pretty flexible.

To see the program's math in action, I wrote up a wrapper script that iterates through different line numbers and probabilities:

demo.sh

#!/bin/sh

for lineno in 1 2 3 4 5 20 100
do
  echo "0 .. 0.99999 < ( 1 / FNR == " $(printf 'scale=2\n1 / %d\n' "$lineno" | bc) ")"
  for r in 0 0.01 0.25 0.5 0.99
  do
    result=$(printf '%f * %d\n' "$r" "$lineno" | bc)
    case $result in
      (0*|\.*) echo "Line $lineno: Result of probability $r * line $lineno is $result and is < 1, choosing line" ;;
      (*)      echo "Line $lineno: Result of probability $r * line $lineno is $result and is >= 1, not choosing line" ;;
    esac
  done
  echo
done

The results are:

0 .. 0.99999 < ( 1 / FNR ==  1.00 )
Line 1: Result of probability 0 * line 1 is 0 and is < 1, choosing line
Line 1: Result of probability 0.01 * line 1 is .010000 and is < 1, choosing line
Line 1: Result of probability 0.25 * line 1 is .250000 and is < 1, choosing line
Line 1: Result of probability 0.5 * line 1 is .500000 and is < 1, choosing line
Line 1: Result of probability 0.99 * line 1 is .990000 and is < 1, choosing line

0 .. 0.99999 < ( 1 / FNR ==  .50 )
Line 2: Result of probability 0 * line 2 is 0 and is < 1, choosing line
Line 2: Result of probability 0.01 * line 2 is .020000 and is < 1, choosing line
Line 2: Result of probability 0.25 * line 2 is .500000 and is < 1, choosing line
Line 2: Result of probability 0.5 * line 2 is 1.000000 and is >= 1, not choosing line
Line 2: Result of probability 0.99 * line 2 is 1.980000 and is >= 1, not choosing line

0 .. 0.99999 < ( 1 / FNR ==  .33 )
Line 3: Result of probability 0 * line 3 is 0 and is < 1, choosing line
Line 3: Result of probability 0.01 * line 3 is .030000 and is < 1, choosing line
Line 3: Result of probability 0.25 * line 3 is .750000 and is < 1, choosing line
Line 3: Result of probability 0.5 * line 3 is 1.500000 and is >= 1, not choosing line
Line 3: Result of probability 0.99 * line 3 is 2.970000 and is >= 1, not choosing line

0 .. 0.99999 < ( 1 / FNR ==  .25 )
Line 4: Result of probability 0 * line 4 is 0 and is < 1, choosing line
Line 4: Result of probability 0.01 * line 4 is .040000 and is < 1, choosing line
Line 4: Result of probability 0.25 * line 4 is 1.000000 and is >= 1, not choosing line
Line 4: Result of probability 0.5 * line 4 is 2.000000 and is >= 1, not choosing line
Line 4: Result of probability 0.99 * line 4 is 3.960000 and is >= 1, not choosing line

0 .. 0.99999 < ( 1 / FNR ==  .20 )
Line 5: Result of probability 0 * line 5 is 0 and is < 1, choosing line
Line 5: Result of probability 0.01 * line 5 is .050000 and is < 1, choosing line
Line 5: Result of probability 0.25 * line 5 is 1.250000 and is >= 1, not choosing line
Line 5: Result of probability 0.5 * line 5 is 2.500000 and is >= 1, not choosing line
Line 5: Result of probability 0.99 * line 5 is 4.950000 and is >= 1, not choosing line

0 .. 0.99999 < ( 1 / FNR ==  .05 )
Line 20: Result of probability 0 * line 20 is 0 and is < 1, choosing line
Line 20: Result of probability 0.01 * line 20 is .200000 and is < 1, choosing line
Line 20: Result of probability 0.25 * line 20 is 5.000000 and is >= 1, not choosing line
Line 20: Result of probability 0.5 * line 20 is 10.000000 and is >= 1, not choosing line
Line 20: Result of probability 0.99 * line 20 is 19.800000 and is >= 1, not choosing line

0 .. 0.99999 < ( 1 / FNR ==  .01 )
Line 100: Result of probability 0 * line 100 is 0 and is < 1, choosing line
Line 100: Result of probability 0.01 * line 100 is 1.000000 and is >= 1, not choosing line
Line 100: Result of probability 0.25 * line 100 is 25.000000 and is >= 1, not choosing line
Line 100: Result of probability 0.5 * line 100 is 50.000000 and is >= 1, not choosing line
Line 100: Result of probability 0.99 * line 100 is 99.000000 and is >= 1, not choosing line

The original formula:

rand() * FNR < 1

can be mathematically rewritten as:

rand() < 1 / FNR

... which is more intuitive to me, as it demonstrates the decreasing values on the right-hand side as the line numbers go up. As the values on the right side of the equation go down, there's a smaller and smaller chance that the rand() function will return a value that's less than the right-hand side.

For each line number, I print a representation of the formula that will be tested: the range of rand()'s output and "1 divided by the line number". I then iterate through some sample random values to see whether the line would be chosen given that random value.

A few sample cases are interesting to look at:

  • on line 1, since rand() generates values in the range 0 <= rand() < 1, the result will always be less than (1 / 1 == 1), so line 1 will always be chosen.
  • on line 2, you can see that the random value needs to be less than 0.50, indicating a 50% chance of choosing line 2.
  • on line 100, rand() now needs to generate a value less than 0.01 in order for the line to be chosen.
  • After looking at this for six to eight minutes, I believe that I understand the math — although I’m not confident that I understand it well enough to explain it.  Do you understand it well enough to explain it, and, if so, would you please do so (or at least link to such an explanation)? – G-Man May 8 at 5:38
  • @G-Man, I've added some explanation – Jeff Schaller May 20 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.