3

I have input piped to awk on a macOS system. I want the sum of two decimal values. One occurs every second row, the other is on every fourth row. I want the result of the sum on every fifth row. The values are delimited by colon+space before, and a comma after. Every sixth row is empty. This pattern repeats until end of the input.

BB: 728345643856359022, 
SH: 3560836, 
RJ: 1500369, 
DD: 1403849, 
Total:,

BB: 729586953244932948, 
SH: 8560836, 
RJ: 4700360, 
DD: 3403021, 
Total:,

BB: 821334125345384020, 
SH: 5293431, 
RJ: 2642499, 
DD: 2433292, 
Total:,

If relevant, which awk outputs /usr/bin/awk and brew list --version outputs gawk 5.2.1_1. Is this possible in g/awk? Any advice appreciated!

To format the input I am using:

getlist | awk '{print $1 $2}' \
| grep 'BB\|SH\|RJ\|DD' \
| sed 's/"//g' \
| awk '{print;} NR % 4 == 0 { print "Total:,";}' \
| awk '{print;} NR % 5 == 0 { print "";}' \
| awk '{print $1 $2}'

Expected output

BB: 728345643856359022, 
SH: 3560836, 
RJ: 1500369, 
DD: 1403849, 
Total: 4964685,

BB: 729586953244932948, 
SH: 8560836, 
RJ: 4700360, 
DD: 3403021, 
Total: 11963857,

BB: 821334125345384020, 
SH: 5293431, 
RJ: 2642499, 
DD: 2433292, 
Total: 7726723,
6
  • 1
    What is the expected output? Commented Feb 27, 2023 at 19:23
  • 1
    I have edited to include expected output.
    – XJMZX
    Commented Feb 27, 2023 at 19:36
  • 1
    and explanation of pipe.
    – XJMZX
    Commented Feb 27, 2023 at 19:37
  • 1
    Should we assume you're on macOS?
    – terdon
    Commented Feb 27, 2023 at 19:40
  • 1
    Yes. I was trying to not specifically limit any help to macOS solutions which is why I highlighted gawk or awk on my system.
    – XJMZX
    Commented Feb 27, 2023 at 19:44

3 Answers 3

3

First, let's simplify your pipeline a little. I don't know what output getlist produces, but based on what you show, this will be equivalent with the minor change of not adding the Total line since that is only making it harder to parse and we will be adding it later:

getlist | 
 tr -d '"' |
 awk '/BB|SH|RJ|DD/{print $1 $2}; NR % 5 == 0 { print ""}' 

You can break lines at |, no need for \ . And you can merge all of the steps into a single awk, and use tr instead of sed for brevity (and likely speed although the difference will be minimal here).

With this in mind, we can extend your awk a little to get the remaining piece:

getlist | 
 tr -d '",' |
 awk '/BB|SH|RJ|DD/{
        print $1 $2","; 
        if(/SH/){ sh = $2 } 
        if(/DD/){ printf "Total: %d,\n", sh + $2 }
      }
      NR % 5 == 0 { print ""}' 

Note that I added , to the characters that tr is removing to facilitate numerical operations (, isn't a number), so I am also adding it back in the various print calls in awk.


In case I have misunderstood, here is a way to do the same thing using the output you already have from your original pipeline (I saved your example as file):

$ awk '{ 
         if(!/Total/){print; 
           if(/SH/){ sh = $2 } 
           if(/DD/){ val = sh + $2 }
         }
         else{
            print "Total:",val","
         }
        }' file
BB: 728345643856359022, 
SH: 3560836, 
RJ: 1500369, 
DD: 1403849, 
Total: 4964685,

BB: 729586953244932948, 
SH: 8560836, 
RJ: 4700360, 
DD: 3403021, 
Total: 11963857,

BB: 821334125345384020, 
SH: 5293431, 
RJ: 2642499, 
DD: 2433292, 
Total: 7726723,
11
  • 1
    Thanks so much. The solution works exactly for my needs. I can see that I could have made the question clearer by showing the input, which may have helped to explain my messy piping. Thanks for the notes about that! One side question. When I ran the code, I substituted sed and tr. tr does indeed seem faster. I'm pretty new to unix commands, what is tr?
    – XJMZX
    Commented Feb 27, 2023 at 21:33
  • 1
    @XJMZX Briefly, it stands for "translate" and is used to, well, translate between one set of characters and another. In this case, I'm using the -d flag which instead deletes characters. It is one of the core utilities, should be available on pretty much any *nix system and, like all *nix tools, is designed to do one thing very fast and very well. See man tr for more. By the way, if one of the answers here helped you, please accept it by clicking on the check mark on the left.
    – terdon
    Commented Feb 27, 2023 at 22:47
  • 2
    Oh, this is JSON!? Yes, there are dedicated tools (e.g. jq) to parse JSON data on the command line and trying to do it with general purpose tools like sed and awk is sometimes easier, but more often is actually harder and is always more fragile and likely to break. So yes, next time, please post the original output when asking.
    – terdon
    Commented Feb 27, 2023 at 22:52
  • 1
    Thanks again! I'll consider doing that if it seems appropriate to @EdMorton
    – XJMZX
    Commented Feb 28, 2023 at 2:12
  • 1
    Yes, that's exactly what it's for.
    – terdon
    Commented Feb 28, 2023 at 9:42
3

Any time you have tag-value pairs in your data as you have, I find it best to first create an array of those mappings (m[] below) and then you can get any value just by indexing the array with the associated name. Using any awk:

$ awk -F': *' -v OFS=': ' '{m[$1]=$2} $1=="Total"{$2=(m["SH"] + m["DD"])","} 1' file
BB: 728345643856359022,
SH: 3560836,
RJ: 1500369,
DD: 1403849,
Total: 4964685,

BB: 729586953244932948,
SH: 8560836,
RJ: 4700360,
DD: 3403021,
Total: 11963857,

BB: 821334125345384020,
SH: 5293431,
RJ: 2642499,
DD: 2433292,
Total: 7726723,
3
  • 1
    That's a very useful explanation. As I commented in other's solutions, I'm learning how to use awk and it's still difficult to test different solutions with pipeline. Thanks for the useful advice.
    – XJMZX
    Commented Feb 27, 2023 at 21:43
  • You're welcome but I can't imagine why input from a pipe command | awk 'script' would be any more difficult to handle than input from a file awk 'script' file.
    – Ed Morton
    Commented Feb 27, 2023 at 22:35
  • Thanks Ed. I was avoiding using read from file until the end of text-processing. But I see your point, there's not a huge mental transformation between the two constructions. Please understand, with every helpful explanation I'm closer to writing my own working scripts :-)
    – XJMZX
    Commented Feb 27, 2023 at 22:57
0

Here's one way, which takes your existing output and achieves the new desired output. Works fine with gawk 5.1.0 here.

$ awk '/^SH/{a=$2}/^DD/{b=$2}/^Total/{$0="Total: "a+b","}1' input.txt
BB: 728345643856359022,
SH: 3560836,
RJ: 1500369,
DD: 1403849,
Total: 4964685,

BB: 729586953244932948,
SH: 8560836,
RJ: 4700360,
DD: 3403021,
Total: 11963857,

BB: 821334125345384020,
SH: 5293431,
RJ: 2642499,
DD: 2433292,
Total: 7726723,
$
3
  • 3
    FYI it'll behave the same way in any awk.
    – Ed Morton
    Commented Feb 27, 2023 at 21:06
  • 1
    Thanks for this alternative solution. I'm unable to get it working with the pipeline. I should have added this more explicitly but I will see if I can get it working and comment again.
    – XJMZX
    Commented Feb 27, 2023 at 21:37
  • 1
    Thatmightbeeasiertoreadifyouusedafewspaces :-)
    – gidds
    Commented Feb 28, 2023 at 12:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .