2

I'm writing a shell script and I'd like to combine the results of the unique values and their counts one file at a time. For example, if I am reading the data from the the two files below (with the number in the first column being the unique count):

Data Read in First Loop

3 Dog   
1 Cat   
5 Horse

Data Read in Second Loop

1 Dog  
3 Cat  
1 Horse  
2 Goat 

I would want the second and first file to be combined in order to overwrite the my "tally file" as:

Resulting File after 2nd Loop Completes

4 Dog  
4 Cat  
6 Horse  
2 Goat 

I'm trying to accomplish this because the data set is large and I would rather read and delete the files than download (or cat the files all together and find the uniq -c at the end since the file would be very long). Is there a way to do this? Thank you!

1

This approach uses awk to do the summing on a tempfile tallytmp and a running tally named simply tally. I'm sorting the two tally files together on the second field (-k2)

It needs a blank file tally to start with, so touch tally, then for each iteration:

uniq -c file > tallytmp
sort -k2 tally tallytmp \
  |awk '{
          if($2==prev){
            tot+=$1
          }else{
            print tot,prev;
            tot=$1;pc=$1;prev=$2
          }
        }END{
          print tot,prev
        }' > tallyresult
cp tallyresult tally

(Go ahead, put the awk script on a single line.)

Logically I should be able redirect the sort|awk pipeline directly to tally, but sort was still using it, so it clobbers it; hence the second tempfile tallyresult before copying that over our results file, tally.

Output of this on the two supplied iterations is actually:

[blank line]
4 Cat
4 Dog
2 Goat
6 Horse

The ordering is mixed up based on ordering of field #2, but it's working correctly. I was able to run a few iterations using this.

0

Someone just upvoted my previous answer and so I had to invent a new solution, of course! This one's a bit better for two reasons:

  1. it's pure bash, using a hash table instead of leaning on awk
  2. you don't get garbage blank lines
  3. it can take an arbitrary number of files to tack onto tally (three reasons)
  4. the tally file can begin empty (four reasons)
  5. it doesn't require a tempfile ("I didn't expect" etc.)

Save the following as tally.sh

#!/bin/bash
# define the associative array tally_table
declare -A tally_table
while read line ; do
    k=($line)
    [ -n "${k[1]}" ] && (( tot[${k[1]}] += ${k[0]} ))
done <<HERE
$(cat tally $@)
HERE
for i in "${!tally[@]}"; do
    echo "${tally[$i]} $i"
done > tally
cat tally

Try removing tally and create a new report file named file with a couple of lines:

1 Horse
3 Monkey

Then feed it in:

$ ./tally.sh file
3 Monkey
1 Horse

Now create another report named newfile with a couple of lines:

5 Horse
2 Pig

And feed it:

$ ./tally.sh newfile
3 Monkey
6 Horse
2 Pig

The Horse entry was correctly incremented. Sure, just as before the ordering isn't consistent. (I don't know how you'd sort for Dog Cat Horse Goat ordering, or where Monkey falls among them.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.