I have a script:


/root/xiotech status > xiostatus.tmp
SyncCount=$(grep -c Sync xiostatus.tmp)
PauseCount=$(grep -c paused xiostatus.tmp)
CopyingCount=$(grep -c Copying xiostatus.tmp)

if [ "$SyncCount" -eq "11" ]
then echo All 11 mirrors are in sync.

else echo $PauseCount mirrors are paused and $CopyingCount mirrors are syncing.

rm -f xiostatus.tmp

Is there a more elegant way to count and "variable-ize" those counts using something like awk? In this case the file is tiny so it's not a big deal, but if the file were 900mb, it would take a lot of extra cycles to go through it 3 times...

  • Can more than one keyword occur on a line? Can the same keyword occur multiple times in one line? – user unknown Feb 14 '18 at 20:07

awk can replace the entire script pretty easily:

#!/usr/bin/awk -f

/Sync/ {SyncCount++}
/paused/ {PauseCount++}
/Copying/ {CopyingCount++}

    if(SyncCount == 11)
        print "All 11 mirrors are in sync."
        print (+PauseCount) " mirrors are paused and " (+CopyingCount) " mirrors are syncing."

The (+var) is to force awk to treat the variable as a number (so it will output 0 if the variable was unset). You can also use a BEGIN block to set all the variables to 0 initially:

    SyncCount = PauseCount = CopyingCount = 0

Stick that in a file and run awk -f /path/to/the/script.awk xiostatus.tmp. If you don't need the temporary file, you can even do /root/xiotech status | awk -f /path/to/the/script.awk.

If you set the execution bit on the awk script, you can call it as a standalone executable: /path/to/the/script.awk xiostatus.tmp, or /root/xiotech status | /path/to/the/script.awk.

  • Works if there is only one instance of the search strings per line. – Arcege Feb 8 '12 at 1:06
  • @Arcege True, I assumed. I don't know if there's a clean way to match the same pattern multiple times per line; I think most people do ugly match()/substr() trickery – Michael Mrozek Feb 8 '12 at 1:12
  • There is only one per line. Is there a way to assign 0 to values that are not returned? the print won't print anything if the value is null, I guess.... – LVLAaron Feb 8 '12 at 1:52
  • @LVLAaron You can use a BEGIN block; I added it to the answer – Michael Mrozek Feb 8 '12 at 1:56
  • You don't need to initialize the vars in BEGIN if you force them to print as numbers and not as (default) strings — just put a plus sign in front of each var and wrap it in brackets (it misbehaves badly with print if you don't use the brackets) — awk 'END{neg=-1; pos=+1; printf "%d %d %d", +neg, +pos, +uninit; print " |", (+neg), (+pos), (+uninit)}' /dev/null — outputs: -1 1 0 | -1 1 0 – Peter.O Feb 8 '12 at 5:21

For those who want to count all instances, here is an awk version which will count multiple non-overlapping instances when there is more than one on the same line

UPDATE: I have now included another method which uses split(.... It is much faster than the match( substr(... method which is now listed below the faster one. The split(... method is more than 4 times faster than the other... (tested on 87 files for a total of 407,612 lines.
For further comparison, Michael Mrozek's method, using /Sync/ range-selection (which counts lines containg each pattern vs. counting all instances of the pattern) is twice as fast as this new method (for the same data).

Another fringe(?) benefit of this faster split(methos) is that it is quite tolerant to invalid UTF-8 chars in the file (so long as they are not in the delimiter pattern)... The delimiters are themselves the actual string patterns being counted... Several of my test files had invalid UTF-8 in them, and it took me quite a while to discover why I got different results from the two methods.
Once the problem files were re-encoded to valid UTF-8, both methods produce identical results.

Here is the new faster method (4+ times faster)... using split(...

awk -v vpat="$pat" 'BEGIN { 
  split(vpat, pat, "|"); for(i in pat) pz++ 
{ if (NF) { for( p in pat ) { ct[p]+=(split( $0, A, pat[p] ) -1) }}
END { print " count   pattern"
      for (p=1; p<=pz; p++) { printf "%6d   %s\n", +ct[p], pat[p] } 
}' file

Here is the Slower method. using match( substr(...

# Count occurrences of multiple non-overlapping string patterns
awk 'BEGIN {
{ for( p in pattern ) {
    while( match( substr( $0, LHB+=(RSTART+RLENGTH-1)), pattern[p] )){
} END {
  print "occurs  pattern"
  for (p in pattern) {
    printf "%6d  %s\n", +count[p], pattern[p] 
}' file

Here is the input file

xx xx  xx
xx             yy           xx

The output is as follows:

occurs  pattern
     5  xx
     1  yy
     0  zz

How about:

eval `/root/xiotech status | grep -Eo 'Sync|paused|Copying' | sort | uniq -c | 
    awk '{print "count_" $2 "=" $1}'`
if [ "$count_Sync" -eq 11 ]; then
    echo All 11 mirrors are in sync.
    echo $count_paused mirrors are paused and $count_Copying mirrors are syncing.

The grep -Eo allows for searching multiple patterns (separated by "|") and returns only the string that matched. The sort | uniq -c shows the count of the words found. The awk script formats new shell commands to create variables starting with "count_". And finally, the eval will take the created shell commands and evaluate them within the shell.

  • This doesn't work. It adds everything up to 11 every time no matter what the lines are. – LVLAaron Feb 8 '12 at 1:53
  • @LVLAaron Really? It works for me, although it has the same problem my answer had (empty strings instead of 0s if the search term doesn't appear) – Michael Mrozek Feb 8 '12 at 1:59
  • @Arcege: +1 It is lightning fast! at 5-6 times faster than the closest contender (Michaek Mrozek's, which I believe can be tweaked to double its speed) ... It just needs some other method of handling patterns which aren't found... There is just one thing about it which prevents this from being general purposs.. It matches the first pattern it finds and then continues the search after that match.. This works for the specs of the question, but it means that because of an overlapping "are", "beware|will|miss|are" and other such overlapping patterns – Peter.O Feb 9 '12 at 3:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.