Using awk to process multiple files need to count occurance of variable after pattern. How can I stop array resetting after each file?

Question

I want to count all occurances of any variable in parenthesis after patterns, for files in a directory that have a specific extension. Each file may contain the pattern on multiple records/line.

So far I can process the files and store the results in an array but seems the array gets overwritten with each new file I process. How can I retain the array value?

#!/bin/bash
for x in `find . $PROGFILES -name "*.fgl"`
do

    awk -f <(cat -  <<-'EOF'
        / envget | env-get | \"envget\" | \"env-get\" /  
        {
        gsub( /get-env/, "envget")              ;# removes hypens
        gsub( /.*envget/, " envget")
        gsub( "\\concat" ,"")       ;# removes concat
        gsub( "\\substring" , "")   ;# removes substring
        for (i = 1; i<= NF; i++) {
            if ( substr( $i, 1, 6) == "envget" ) {
                    lenofget = 8;
                } else {
                    lenofget = 0;
                }
                if ( lenofget != 0 ) {
                    gsub("\\envget" , "",$i)    ;#removes envget
                    gsub ( /\)\.*/, "",$i)      ;#removes everything after a closing parenthesis
                    gsub ( /\47/, "",$i)        ;#so used octal instead
                    gsub ( /\(/, "",$i)         ;#removes paraentheses
                    gsub ( /\"/, "",$i)         ;#removes double quotes
                    gsub ( /\,.*/, "",$i)       ;#removes everything after a , This is for any concat syntax
                    gsub ( /[\/].*/, "",$i)     ;#removes everything after a forward slash
                    narr[$i]++
                }
            }
        }
        END {
            for (y in narr) {
                printf("%s - %d\n",y, narr[y])

            }   
        }   
EOF
) $x
done

Typical records/lines in a file with the pattern/s would be:

if envget("SYPSDATA") in {SPACES "."}
    set lf-path = "envget"('SYPSCTRL')
if env-get(concat("LOG_PRINTER",service-centre)) != spaces
trconcat(env-get("TMPDIR"),"/ps_xxx_temp.psv")
envget(substring(ws-envprinter1,1,strlen(ws-envprinter1)))
      set lf-path = "envget"('SYPSCTRL')
            display bitmap concat(envget('BTS')'/images/repedge.gif') @19,44

Given there are multiple files with multiple pattern matched lines I'm expecting to get output like this (where the numbers are total count found in every file).

BTS - 15
LOG_PRINTER - 7
ws-envprinter1 - 3
SYPSDATA - 120
TMPDIR - 130
SYPSCTRL - 200

Answer 1

You're doing

for x in `find . $PROGFILES -name "*.fgl"`
do
    awk (awk_program) $x
done

which starts a new awk process for each file.  Why?  Just do

awk (awk_program) *.fgl "$PROGFILES"/*.fgl

unless you need to search subdirectories.  If you do need to search subdirectories, it’s only a little more complicated:

find . "$PROGFILES" -name "*.fgl" -exec awk (awk_program) {} +

Notes:

• You should always quote shell variables (like "$PROGFILES" and "$x") unless you have a good reason not to, and you’re sure you know what you’re doing.

• You don’t need to use a cat like that.  You can put the awk program in quotes:

  awk '
          / envget | env-get | \"envget\" | \"env-get\" /  
          {
              gsub( /get-env/, "envget")
                          ︙
          }
      ' "$x"
  

  or you can put it into a file and say awk -f (awk_program_file).

• Neither of the above approaches is guaranteed to get the total counts because there is a (very large) limit on the size of a command line.  If you have so many files that the combined length of their names exceeds that limit, find will invoke multiple awk processes to cover all the names, and you’ll be back to getting incomplete counts.  One way to handle this would be to collect the outputs from the individual awk runs and combine them.

Answer 2

There are several mistakes in your attempt (not trying to be condescending, we are all here to learn !).

Every awk invocation is a different process, with its own memory space, so once a file has been processed, it is normal that the next awk invocation does not retain the value for your array. You need to output count at every iteration of your for, and an extra step at the end to sum it all up. The easiest way is to add all this to a file:

#!/bin/bash

echo "" > "$HOME/tmp_count.txt"

for x in `find . $PROGFILES -name "*.fgl"`
do

    awk '
        /env-?get/  {
          for (i = 1; i<= NF; i++) {
            if ($i ~ /env-?get/) {
              a = gensub(/.*env-?get\"?\((concat\(|substring\()?(\"|\47)?([a-zA-Z0-9\-_]*)(\"|\47)? *(\)|,)?.*/, "\\3", $i)
              arr[a]++
            }
          }
        }
        END {
            for (y in arr) {
                printf("%s %d\n",y, arr[y])
            }   
        }
        ' "$x" >> "$HOME/tmp_count.txt"
done

awk '{arr[$1] += $2}END{for (key in arr) {printf("%s - %d\n", key, arr[key])}}' < "$HOME/tmp_count.txt"

Also your method seems to not work all the time, so I tried using one big regexp in a gensub, it works at least with the sample you provided. I'm no regexp wizard though so it might break on certain occurences. Try it out and let me know if it worked out for you !