Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a file like this:

A 100
A 200
A 300 #sum=600
B 400
B 500 #sum=900
A 600
A 700
A 800 #sum=2100

I would like the output to be:

A 600
B 900
A 2100
C sum_of_C
D sum_of_D

I can do that with for, sed, grep and awk.

However because I am learning awk, I would like to write an awk script. So far I have:

if (${NR {print $1}} == ${NR-1 {print $1}}) 
  sum+=$2
  print $0"\t"sum
else
  sum=$2
  print $0"\t"sum

awk -f awkscript file was not successful. What is the solution?

share|improve this question

2 Answers 2

up vote 6 down vote accepted

I'm not completely sure what your if is trying to do there. NR is the number of records; use NF for the number of fields, if that's what you're aiming for. You can't put {} blocks in the middle of things like that.

I think what you're aiming for is to compare the value of a field in this line with a field in the previous line, printing out the sum when we reach a new "group" of data. If that's the case, this script will do what you want and I think equates pretty much to what you were aiming for:

{
    if (last && $1 != last) {
        print last, sum
        sum = 0
    }
    sum = sum + $2
    last = $1
}
END {
    print last, sum
}

We make a new variable last to hold the value of the first field ($1) on the previous line. We'll use that to track which group we're looking at.

  • For every line (because we have { ... } at the top level), we first test whether a) last is set (because we don't want to print anything on the very first line), and b) the value of the first field is different than last. If it is, we print out the value of last, a space (because of ,), and the sum we've calculated. (If you want a tab, use "\t" in quotes like you had)
  • After printing, we reset sum to zero.
  • Either way, we add the value of the second field ($2) to sum.
  • For every line, we save the first field (our group) into last, so we can use it for comparison on the next line.
  • Finally, we want to print out the last group as well. For that, we use an END { ... } block. It runs right at the end of the program when we run out of data. We print out the sum and the group we're working with just like we did before.

If I run:

awk -f sum.awk < data

with your data file, I get this output:

A 600
B 900
A 2100

as desired.


There are simpler ways to do this, both in awk and otherwise. In particular, we can replace the body above with:

last && $1 != last {
    print last, sum
    sum = 0
}
{
    sum = sum + $2
    last = $1
}

Here we use awk's conditional block syntax rather than an explicit if test: the behaviour of this program is identical to the one above, but it's more idiomatic. It's not hugely different in this example, but it's useful to know about if you're learning awk.


If the file example you gave is literally what it is, with #sum= lines (or similar), you can use this script:

{
    sum = sum + $2
    if (NF == 3) {
        print $1, sum
        sum = 0
    }
}

For every line, this adds the value of the second field to the sum variable. On lines that have exactly three fields (NF == 3), we print out our total, and reset sum to zero.

share|improve this answer
    
+1000. It works like a charm. I didn't expect to have an answer with detail explanations so quick. Thank you so much for your effort. May I ask you one thing? Why do me need to have last && $4 != last? Isn't only $4 != last enough? And last isn't a boolean anyway, it's a string. –  Ooker Jul 7 at 10:28
    
On the very first line, last is unset and so is always different than $1. We don't want to treat that as the end of a group, so the test is there as a special case for the very first line. Used as a boolean an empty string is false. –  Michael Homer Jul 7 at 10:33
    
If last is unset, then sum=sum+$2=$2 as expected. I can't see what's wrong. –  Ooker Jul 7 at 10:51
    
Try taking out last && and see what happens. –  Michael Homer Jul 7 at 21:28
    
The point is, I have tried and nothing changes –  Ooker Jul 7 at 21:31

If your file is small enough for all sums to fit in memory, you could do something as simple as:

$ awk '{sum[$1]+=$2}END{for(pat in sum){print pat,sum[pat]}}' file 
A 2700
B 900

Here's the same thing as a commented awk script:

#!/usr/bin/awk -f

{
    ## Here, we use $1 as the key of an associative array
    ## and increment its current value by $2. The result of 
    ## this will be an array element for each different $1 in 
    ## the file whose value will be the sum of all associated $2s.
    sum[$1]+=$2
}

## The END{} block is exacuted after the entire file
## has been processed.
END{
    ## Iterate through the keys of the array (the $1s),
    ## saving each as 'pat'. Then, print the current value of
    ## 'pat' as well as the associated value (the sum) from
    ## the array.
    for(pat in sum){
        print pat,sum[pat]
    }
}

The only possible problem with this approach is if you have so many lines that keeping the array of $1s will cause you to run out of memory. This is not very likely on a modern system. On the other hand, this approach also works when your file's lines are not in order since it can deal with unsorted files.

share|improve this answer
    
Thank you. Do you know how many lines are considered to be too much? –  Ooker Jul 7 at 10:53
1  
@Ooker that depends on your hardware. For this kind of thing, any modern system should be able to deal with many millions. However, note that this prints the output of all As and then all Bs. I just re-read your question and it looks like that's not what you want. If you want to print the sum of each group of As, Bs, etc., you should use Michael's approach. –  terdon Jul 7 at 10:55
    
Yes, you're right. However I do need to sum all A, then I do need your answer. –  Ooker Jul 7 at 10:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.