I would like to run a for-loop, which computes the following task: Run awk 10 times, sum output and print final result. The code is attached below:

for i in {1..10}
  count=`awk '{if ($NF==$i) {print $NF}}' * | wc -l`
  echo $sum

The issue is, if I change $NF==$i to $NF==1, then the result is correct, but I would like to use the for-loop to run 10 times.

What is the issue in the code?


The issue here is that you are trying to get shell variable expansion inside single quotes ' ... '. However, inside single quotes, shell variable expansion is suspended (see e.g. this answer here or Lhunath & GreyCat's Bash Guide), which is the reason why this is actually recommended: as the $ performs a similar function in awk in dereferencing individual input fields (but where the field number can be expressed by a variable name as well, as in e.g. $NF), enclosing the program in single quotes avoids concurrent "variable expansions".

In your case, you can "import" the value into the awk program with

awk -v fieldnr="$i" '{if ($NF==fieldnr) {print $NF}}'

Still, it would appear that your problem can be solved entirely in awk, so maybe you want to explain what you want to accomplish in more detail and we can try to find a more elegant (and possibly faster) way; it may be reasonable to open another question, though ...

  • Thanks for your reply. The intention is to run a for loop, each loop do awk '{if ($NF==$i) {print $NF}}' * | wc -l, where i from 1 to 10. I find that change to count=awk -v fieldnr=$i '{if ($NF==fieldnr) {print $NF}}' * | wc -l does the job. Your solution is very helpful! – 9f241e21 Feb 18 at 13:03
  • 1
    @9f241e21 You don't need a shell loop since awk can do the whole thing on it's own, and even if you did need a loop for some reason you don't need to pipe the awk output to wc since awk can count lines for itself too and given that you don't need awk to print $NF. Also, awk is made up of condition { action } statements so you shouldn't put an explicit condition inside the action part while leaving an implicit true condition to execute it. You got an answer to the question you asked but if you'd like help to do whatever it is you're trying to do the right way then post a new question. – Ed Morton Feb 18 at 18:02
  • 1
    @EdMorton Thanks for your reply. I agree that my solution is very inefficient. I'll check based on your suggestions – 9f241e21 Feb 19 at 0:55

It would be more efficient to do the whole thing in awk:

awk -v n=0 '
  {for (i = 1; i <= 10; i++) if ($NF == i) {n++; break}}
  END {print n}' ./*

Or if the last field is always an integer:

awk -v n=0 '
  $NF >= 1 && $NF <= 10 {n++}
  END {print n}' ./*

Or if you want the comparison to be lexical and only allow the 1, 2, 3..., 10 representations of those numbers (not 01, 1e0, 1.0...):

awk -v n=0 '
  $NF ~ /^([123456789]|10)$/ {n++}
  END {print n}' ./*
  • Pretty neat solution. Thanks! – 9f241e21 Feb 18 at 13:17
  • Nbd but if you change print n to print n+0 then you don't need the -v n=0 up front. – Ed Morton Feb 18 at 18:11

If your script does what I think it does (i.e. print incremental values of sum for the matching lines as it loops from 1 to 10) then all you really need to produce the output your shell script would produce is:

awk '
{ counts[$NF]++ }
    for (i=1; i<=10; i++) {
        sum += counts[i]
        print sum
' *

The above is untested since you didn't provide any sample input/output to test against.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.