I'm trying to find derive SCALEFACTOR which is basically 10000/(sum of 4th column in a file). How do I get a decimal from the output? Appreciate any help in advance.


for f in ${FILES}
    echo $f
    COLTOTAL="$(awk '{sum += $4} END {print sum}' $f)"
    echo "total: ${COLTOTAL}"
#    SCALEFACTOR=`expr 10^5 / $COLTOTAL`
    SCALINGFACTOR=$(echo "100000 / $COLTOTAL" | bc -l
#    echo "scale=5; ${SCALEFACTOR}" | bc
    echo ${SCALEFACTOR}
    awk '{print($1"\t"$2"\t"$3"\t"$4 * ${SCALINGFACTOR})}' $f > $f"_normalized.txt"

  • 3
    bash only does integer math. Just use awk and be done in one line: awk '{sum += $4} END {print 10^5/sum}' $f – jw013 Feb 24 '15 at 18:54
  • Your problem here is that you're performing the (integer) division in the shell, and then asking bc to just print the number. You want quotes SCALEFACTOR=$(echo "$B / $COLTOTAL" | bc -l not arithmetic parentheses $(( )) – glenn jackman Feb 24 '15 at 19:30
  • Thank you. This is a better option in my case. How do I use SCALEFACTOR variable if I want to multiply it to another variable? – Stephen Feb 24 '15 at 20:58

The way you implemented your for loop will break if you have files with spaces in them. It will work just fine without the variable, for f in /path/to/files/*, since the expansion happens in a manner that the for loop can understand.

Generally, since it does cost you resources to spin up new processes, it is best to have just one instance of awk and, as @jw013 noted, you must perform the division outside of the shell since sh and bash are incapable of floating point math.

Because you need knowledge of the end of the file to manipulate each line, you have two options: read through each file, saving each line for the second readthrough, or read each file twice. Since saving large files in memory can be problematic, I've opted for the second option:

for f in /path/to/files/*; do
  echo "$f"
  awk '
    NR == FNR {
      sum += $4;
    FNR == 1 {
      print "total: " sum;
      SCALEFACTOR = 10000 / sum;
      print SCALEFACTOR;
      printf("%s\t%s\t%s\t%f\n", $1, $2, $3, $4 * SCALEFACTOR);
    }' "$f" "$f"

NR == FNR says the overall record (line) number is the same as the record number of the current file, which means you're on the first file and the task at hand is determining the sum. next prevents the other clauses from firing. Otherwise, if on the first line of the second reading of the file, we do the stuff you had between your awk calls. For each line in that second reading, we print the four items, with the fourth item scaled as you indicated.

You can use standard string formatting on that fourth item, e.g. your commented bc scale of five would change my %f to %.5f

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.