1

I want to count all occurances of any variable in parenthesis after patterns, for files in a directory that have a specific extension. Each file may contain the pattern on multiple records/line.

So far I can process the files and store the results in an array but seems the array gets overwritten with each new file I process. How can I retain the array value?

#!/bin/bash
for x in `find . $PROGFILES -name "*.fgl"`
do

    awk -f <(cat -  <<-'EOF'
        / envget | env-get | \"envget\" | \"env-get\" /  
        {
        gsub( /get-env/, "envget")              ;# removes hypens
        gsub( /.*envget/, " envget")
        gsub( "\\concat" ,"")       ;# removes concat
        gsub( "\\substring" , "")   ;# removes substring
        for (i = 1; i<= NF; i++) {
            if ( substr( $i, 1, 6) == "envget" ) {
                    lenofget = 8;
                } else {
                    lenofget = 0;
                }
                if ( lenofget != 0 ) {
                    gsub("\\envget" , "",$i)    ;#removes envget
                    gsub ( /\)\.*/, "",$i)      ;#removes everything after a closing parenthesis
                    gsub ( /\47/, "",$i)        ;#so used octal instead
                    gsub ( /\(/, "",$i)         ;#removes paraentheses
                    gsub ( /\"/, "",$i)         ;#removes double quotes
                    gsub ( /\,.*/, "",$i)       ;#removes everything after a , This is for any concat syntax
                    gsub ( /[\/].*/, "",$i)     ;#removes everything after a forward slash
                    narr[$i]++
                }
            }
        }
        END {
            for (y in narr) {
                printf("%s - %d\n",y, narr[y])

            }   
        }   
EOF
) $x
done

Typical records/lines in a file with the pattern/s would be:

if envget("SYPSDATA") in {SPACES "."}
    set lf-path = "envget"('SYPSCTRL')
if env-get(concat("LOG_PRINTER",service-centre)) != spaces
trconcat(env-get("TMPDIR"),"/ps_xxx_temp.psv")
envget(substring(ws-envprinter1,1,strlen(ws-envprinter1)))
      set lf-path = "envget"('SYPSCTRL')
            display bitmap concat(envget('BTS')'/images/repedge.gif') @19,44

Given there are multiple files with multiple pattern matched lines I'm expecting to get output like this (where the numbers are total count found in every file).

BTS - 15
LOG_PRINTER - 7
ws-envprinter1 - 3
SYPSDATA - 120
TMPDIR - 130
SYPSCTRL - 200
1

You're doing

for x in `find . $PROGFILES -name "*.fgl"`
do
    awk (awk_program) $x
done
which starts a new awk process for each file.  Why?  Just do

awk (awk_program) *.fgl "$PROGFILES"/*.fgl

unless you need to search subdirectories.  If you do need to search subdirectories, it’s only a little more complicated:

find . "$PROGFILES" -name "*.fgl" -exec awk (awk_program) {} +

Notes:

  • You should always quote shell variables (like "$PROGFILES" and "$x") unless you have a good reason not to, and you’re sure you know what you’re doing.
  • You don’t need to use a cat like that.  You can put the awk program in quotes:

    awk '
            / envget | env-get | \"envget\" | \"env-get\" /  
            {
                gsub( /get-env/, "envget")
                            ︙
            }
        ' "$x"
    

    or you can put it into a file and say awk -f (awk_program_file).

  • Neither of the above approaches is guaranteed to get the total counts because there is a (very large) limit on the size of a command line.  If you have so many files that the combined length of their names exceeds that limit, find will invoke multiple awk processes to cover all the names, and you’ll be back to getting incomplete counts.  One way to handle this would be to collect the outputs from the individual awk runs and combine them.
  • needs to search subdirectories as well – BoRain Nov 14 '17 at 21:59
  • In case a lot of files are present, the + version of find's exec will process the files in batches (can't remember how many files at a time but not that many), so I don't think it will work. The man page for find states: "the total number of invocations of the command will be much less than the number of matched files". So it is definitely not guaranteed that only one invocation will be used, but it can definitely speed up such commands. – Valentin B. Nov 15 '17 at 9:03
  • Also if you have a lot of lengthy files, not sure loading all their content at once into your memory is a fine idea. – Valentin B. Nov 15 '17 at 9:08
0

There are several mistakes in your attempt (not trying to be condescending, we are all here to learn !).

Every awk invocation is a different process, with its own memory space, so once a file has been processed, it is normal that the next awk invocation does not retain the value for your array. You need to output count at every iteration of your for, and an extra step at the end to sum it all up. The easiest way is to add all this to a file:

#!/bin/bash

echo "" > "$HOME/tmp_count.txt"

for x in `find . $PROGFILES -name "*.fgl"`
do

    awk '
        /env-?get/  {
          for (i = 1; i<= NF; i++) {
            if ($i ~ /env-?get/) {
              a = gensub(/.*env-?get\"?\((concat\(|substring\()?(\"|\47)?([a-zA-Z0-9\-_]*)(\"|\47)? *(\)|,)?.*/, "\\3", $i)
              arr[a]++
            }
          }
        }
        END {
            for (y in arr) {
                printf("%s %d\n",y, arr[y])
            }   
        }
        ' "$x" >> "$HOME/tmp_count.txt"
done

awk '{arr[$1] += $2}END{for (key in arr) {printf("%s - %d\n", key, arr[key])}}' < "$HOME/tmp_count.txt"

Also your method seems to not work all the time, so I tried using one big regexp in a gensub, it works at least with the sample you provided. I'm no regexp wizard though so it might break on certain occurences. Try it out and let me know if it worked out for you !

  • Thanks you've put me on the right track. Couldn't get the gensub working right but did as you said and added to a file. Only stored string in file (no count of 1) because this produced a string 1 - count eg.SYSPCTRL 1 - 2 – BoRain Nov 14 '17 at 21:57
  • Yes I have made a regexp that works on my side for the example you provided but I assume it might break for certain occurences in your files. if you find that your method works in most cases, it's all good. Your main problem was the multiple awk invocations. – Valentin B. Nov 15 '17 at 8:51
  • Also if you follow G-Man's advice and put your awk code into a file (say my_awk_script.awk), you can even get rid of the for loop (find will do it for you): find . "$PROGFILES" -name "*.fgl" -exec awk -f my_awk_script.awk "{}" \;, and then treat your temporary file with the final awk command in my answer. – Valentin B. Nov 15 '17 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.