Variable Capture and For-Loop Arithmetic

Question

My input files are a single column with variable length of rows. My code is supposed to count those rows, and print a specified number. The problem I'm having, is if the number of rows captured in the variable "A" is less than 1, I want it to print at least 1 row so the output file isn't empty. If "A" (1% of the total number of rows) is greater than 1, I want to print that number of rows. My hybrid awk-bash code looks like this:

#!/bin/sh
for i in {1..2}
do
input="../ExpressionSet_"$i"_chunk.txt"
    for j in {1..2}
do

A=$(awk 'END{print NR*0.01}' $input)
Y=1
X=0
if (( "$A" -lt "$Y")); then
X=$A+1
else
X=$A

fi
awk 'NR<='$X' {print $0}' $input > "$i"_top1pc.txt

B=$(awk 'END{print NR*0.05}' $input) 
awk 'NR<='$B' {print $0}' $input > "$i"_top5pc.txt

Confusingly, I keep getting error messages like

thresholdSelector_pc.sh: line 20: ((: 0.24 -lt 1: syntax error: invalid arithmetic operator (error token is ".24 -lt 1")
thresholdSelector_pc.sh: line 20: ((: 47.24 -lt 1: syntax error: invalid arithmetic operator (error token is ".24 -lt 1")

By the way, inputFile1 has 24 lines and inputFile2 has 4724 lines. Thanks for the help!

Answer 1

I think your root problem is that you tried to use bash syntax in an sh script. Bash defines extensions to the sh common denominator; if you want to use bash-specific features, your script must start with #!/bin/bash, not #!/bin/sh.

Sh doesn't have the ((…)) syntax for arithmetic expressions. But you don't need it here, you can use the portable [ … ] conditional syntax. In the [ … ] conditional, the “less than” operator is written -lt.

Braces {1..2} are another bash feature that doesn't exist in sh. Another bug in your script is X=$A+1, which sets X to a string like 42+1 if the value of A is 42; to perform an arithmetic calculation, you need to use the arithmetic expression syntax $((…)).

Also, as a general remark, always use double quotes around variable substitutions.

Another problem with your code is that it looks like A is a decimal number. Shell arithmetic only works with integers. I've adapted the algorithm but check what it does, I might not have rounded the way you intended. Using awk just to count the number of lines is overkill, wc -l is a clearer and faster way of doing that. Similarly, to print the first N lines of a file, just call head.

Yet another bug is that $i_chunk is the value of the variable i_chunk. To take the value of i and append _chunk, you need to delimit the variable name with braces: ${i}_chunk.

I have no idea what the loop on j is supposed to be doing, I've left it alone.

#!/bin/sh
for i in 1 2
do
  input="../ExpressionSet_${i}_chunk.txt"
  for j in 1 2
  do
    A=$(wc -l <"$input")
    Y=100
    X=0
    if [ "$A" -lt "$Y" ]; then
      X=$((A+100))
    else
      X=$A
    fi
    head -n "$((X/100))" "$input" > "$i"_top1pc.txt
    head -n "$((X/20))" "$input" > "$i"_top5pc.txt
  done
done

If you choose to write a bash script, you can make use of several bash features:

• ((…)) for arithmetic evaluations (but still integer only)
• typeset -i to declare an integer variable, so that assigning an arithmetic expression evaluates it

#!/bin/bash
for i in 1 2
do
  input="../ExpressionSet_${i}_chunk.txt"
  for j in 1 2
  do
    A=$(wc -l <"$input")
    Y=100
    typeset -i X
    if ((A < Y)); then
      X=A+100
    else
      X=$A
    fi
    head -n "$((X/100))" "$input" > "$i"_top1pc.txt
    head -n "$((X/20))" "$input" > "$i"_top5pc.txt
  done
done