I know there are "sort" and "uniq" out there, however, today's question is about how to utilise AWK to do that kind of a job. Say if I have a list of anything really (ips, names, or numbers) and I want to sort them;

Here is an example I am taking the IP numbers from a mail log:

awk 'match($0,/\[[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+\]/) { if ( NF == 8 && $6 == "connect" ) {print substr($0, RSTART+1,RLENGTH-2)} }' maillog

Is it possible to sort them, ips, "on the go" within the same awk command? I do not require a complete answer to my question but some hints where to start.



To sort you can use a pipe also inside of an awk command, as in:

awk '{ print ... | "sort ..." }'

The syntax means that all respective lines of the data file will be passed to the same instance of sort.

Of course you can also do that equivalently on shell level:

awk '{ print ... }' | sort ...

Or you can use GNU awk which has a couple sort functions natively defined.

The uniq is in awk typically accomplished by saving the "unique data element or key" in an associative array and checking whether new data need to be memorized. One example to illustrate:

awk '!a[$0]++'

This means: If the current line is not in the array then the condition is true and the default action to print the line triggered. Subsequent lines with the same data will result in a false condition and the data will not be printed.

  • 4
    Strictly speaking, !a[$0]++ is not equal to uniq, because uniq requires input data sorted. You need awk 'l != $0 {l = $0}'. – cuonglm Mar 30 '15 at 8:53
  • 3
    No, it's of course not equal. uniq has a couple behaviours that you can choose depending on the options. But the OP asked for something "to do that kind of a job"; i.e. how to accomplish that in principle with awk. The method I presented is in this respect more powerful since its logic operates over all data in the file. This is on Unix typically reflected by some-process | sort | uniq, or (with a different semantic) by some-process | sort -u. Again, this is also not "equal", but you rarely want to imitate a Unix command, but rather solve a conncrete task. – Janis Mar 30 '15 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.