9

[edit: clarified that I need an in awk solution, and corrected that I need to sort 'indexes' (or rather, output them in a sorted way) instead of the ambiguous 'values')]

In awk, I often count things, or store a set of values, inside an array, using the values as indices (taking advantage of awk's indexes_are_hashes mechanism)

For example: if I want to know how many different values of $2 I encountered, and how often each values were seen:

awk '
   ... several different treatments ...
   { count[$2]++ } 
   ... other treatments ...
   END { for(str in count) { 
           print "counted: " str " : " count[str] " times." 
           ... and other lines underneath, with additional infos ...
          }
       }
 '

The problem is that (non GNU, or other nicer versions) regular awk (and regular nawk) :

  • [A] doesn't output the different values in the order it has 'encountered' them,
  • [B] nor provide an easy way to go through the indexes either in numerical or alphabetical order

for [A]: not too difficult to do .. just have another array to index the "newly seen" entries.

the QUESTION is for [B]: How can I do a simple call to sort to reorder the display of the different indexes?

(note : I am aware that gnu awk has an "easy" way for [B]: https://www.gnu.org/software/gawk/manual/html_node/Controlling-Array-Traversal.html ... But I want the way to do something similar in regular awk/nawk !)

(ie: I need to do a loop to output the different indexes seen, sort them, re-read them [in an old awk...] into "something" ( ex: another array ordered_seen ?) and use that something to display the seen[s] in the chosen order. And this needs to be inside awk as under each indexes I often need to output a paragraph of additional infos. A "sort" outside of awk would reorder everything)

So far: I find no "axiomatic" one-liner (or n-liner?) way to do that.

I end up with a kludge that takes several lines, outputs each values to a file through sort, and then re-reads that sorted file and insert each line in order into a sorted_countindexes[n++], and then for(i=0;i<=n;i++){ ...output count[sorted_countindexes[n]]... }

I'd welcome a better/simpler/more "axiomatic" to output indexes according to a sort, for regular awk (or nawk)

MCVE: here is a simple example : outputting the indexes in alphabetical order would be really nice:

# create the 2 basic files to be parsed by the awk:
printf 'a b a a a c c d e s s s s e f s a e r r f\ng f r e d e z z c s d r\n' >fileA
printf 's f g r e d f g e z s d v f e z a d d g r f e a\ns d f e r\n'>fileB
# and the awk loop: It outputs in 'whatever order', I want in 'alphabetical order'
for f in file? ; do printf 'for file: %s: ' "$f"
  tr ' ' '\n' < "$f" | awk ' 
       { count[$0]++ } 
   END { for(str in count){ 
           printf("%s:%d ",str,count[str]) 
          }; print "" 
       } '
done
#this outputs:
for file: fileA: d:3 e:5 f:3 g:1 r:4 s:6 z:2 a:5 b:1 c:3
for file: fileB: d:5 e:5 f:5 g:3 r:3 s:3 v:1 z:2 a:2
# I'd like to have the letters outputted in alphabetical order instead!
3
  • 1
    @EdMorton I updated the question to explain why I need an "inside awk" solution to order the indexes (not the values as I wrongly said). iow: for your 'a)' I can't as I use the indexes to output "paragraphs" of informations, with different kind of infos on each paragraph lines. I need b) or c). And I'll try to find a few different MCVEs as I need a generic solution (not just for 1 case) Sep 17, 2020 at 13:35
  • 1
    @EdMorton I added an MCVE at the end Sep 17, 2020 at 14:24
  • It was a tough decision to decide which answer to accept : both are good. @EdMorton's better now as it is a bit more readable, and versatile (as it uses "sort"), but Stephane's answer is very good too (and doesn't need external commands) Sep 25, 2020 at 12:53

2 Answers 2

5

With GNU awk, you could do a two-way interaction with sort with its "coprocess" feature (info gawk coproc), where you'd send the data to sort with print |& "sort" and get the result with "sort" |& getline, but again that's gawk specific.

To loop over the array in the order it's been encountered, you could record that encounter sequence at the time you fill in the array:

awk '
  !seen[$1]++ {sequence[n++] = $1}
  END {
    for (i = 0; i < n; i++)
      print sequence[i], seen[sequence[i]]
  }'

You could also implement the sorting algorithm in awk. You could even borrow gawk's quicksort.awk, you'll even find it in its manual (here to demonstrate indirect function calls, another GNU-specific feature, you'd replace that with a literal call to your comparison routine). Something like:

awk '
  function less_than(left, right) {
    return "" left <= "" right
  }
  function quicksort(data, left, right,   i, last)
  {
    if (left >= right)
      return

    quicksort_swap(data, left, int((left + right) / 2))
    last = left
    for (i = left + 1; i <= right; i++)
      if (less_than(data[i], data[left]))
        quicksort_swap(data, ++last, i)
    quicksort_swap(data, left, last)
    quicksort(data, left, last - 1)
    quicksort(data, last + 1, right)
  }
  function quicksort_swap(data, i, j,   temp)
  {
    temp = data[i]
    data[i] = data[j]
    data[j] = temp
  }

  {seen[$1]++}
  END {
    for (i in seen) keys[n++]=i
    quicksort(keys, 0, n-1)
    for (i = 0; i < n; i++)
      print keys[i], seen[keys[i]]
  }'

Personally, I'd just use perl instead of awk here.

0
3
$ cat tst.awk
{ cnt[$0]++ }
END {
    n = sort(cnt,idxs)
    for (i=1; i<=n; i++) {
        idx = idxs[i]
        printf "%s:%d%s", idx, cnt[idx], (i<n ? OFS : ORS)
    }

}

function sort(arr, idxs, args,      i, str, cmd) {
    for (i in arr) {
        gsub(/\047/, "\047\\\047\047", i)
        str = str i ORS
    }

    cmd = "printf \047%s\047 \047" str "\047 |sort " args

    i = 0
    while ( (cmd | getline idx) > 0 ) {
        idxs[++i] = idx
    }

    close(cmd)

    return i
}

# create the 2 basic files to be parsed by the awk:
printf 'a b a a a c c d e s s s s e f s a e r r f\ng f r e d e z z c s d r\n' >fileA
printf 's f g r e d f g e z s d v f e z a d d g r f e a\ns d f e r\n'>fileB

for f in fileA fileB ; do
    printf 'for file: %s: ' "$f"
    tr ' ' '\n' < "$f" |
    awk -f tst.awk
done
for file: fileA: a:5 b:1 c:3 d:3 e:5 f:3 g:1 r:4 s:6 z:2
for file: fileB: a:2 d:5 e:5 f:5 g:3 r:3 s:3 v:1 z:2

The above just builds a newline-separated string from the array indices (quoting it appropriately for sh), creates a shell script that pipes that string to sort, and then loops on the output. If you want to modify sorts behavior just add a string of Unix sort arguments to the sort function call, e.g. sort(seen,"-fu"). It could obviously be modified to print or do whatever else you want inside the sort() function instead of populating an array of indices for you to loop on when it returns if that's what you prefer but then the function is as cohesive.

Note however that it will be limited to the maximum command line length on your system.

The \047s in the code represent 's which shell does not allow to be included in '-delimited strings or scripts and so while we could use ' directly in an awk script being read from a file as I'm doing above, if you were to use that script on the command line as awk 'script' file you'd need to use something instead of ' and \047 works both when the script is interpreted from the command line and from a file so it's the most portable choice of '-replacement.

The 's (\047s) are present to quote str in a way that ensures that the shell doesn't expand variables, have mismatched quotes, etc. when the string is being piped to sort, i.e. they do this:

$ echo 'foo'\''bar $(ls) $HOME' | awk '{
    str=$0; gsub(/\047/, "\047\\\047\047", str); print "str="str
    cmd="printf \047%s\047 \047" str "\047"; print "cmd="cmd
}'
str=foo'\''bar $(ls) $HOME
cmd=printf '%s' 'foo'\''bar $(ls) $HOME'

so we don't get something like this, which is vulnerable/buggy, instead:

$ echo 'foo'\''bar $(ls) $HOME' | awk '{
    str=$0; print "str="str
    cmd="printf \"%s\" \"" str "\""; print "cmd="cmd
}'
str=foo'bar $(ls) $HOME
cmd=printf "%s" "foo'bar $(ls) $HOME"
21
  • 1
    I agree for "seen", I just reused the name by simplifying a long awk script using seen as such, to the MCVE exemple which really uses the count themselves. my bad. Sep 17, 2020 at 14:33
  • 1
    oh, i see, args is a placeholder for eventuel additional arguments to sort. quite nice, indeed. seems quite generic. thanks for your input Sep 17, 2020 at 14:35
  • 1
    @OlivierDulac when you say count(), you mean sort(), right? Declared function arguments that aren't passed in when the function is called (e.g. i, str, cmd) are treated as function-local variables and as such are initialized to zero-or-null each time the function is called. That's the syntax for using local variables in awk - a variable named i outside of the function would be a different variable to i inside of it, for example.
    – Ed Morton
    Sep 18, 2020 at 12:13
  • 1
    No worries. Not sure why you'd use \047 in place of ' though (note that ' is only 047 on ASCII based systems). Note that % is not special in any sh implementation, it doesn't need quoted. Sep 18, 2020 at 13:19
  • 1
    Ed, thanks a lot for this, I didn't knew that (... I really should read the o'reilly awk book...) Sep 18, 2020 at 13:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .