39

This one-liner removes duplicate lines from text input without pre-sorting.

For example:

$ cat >f
q
w
e
w
r
$ awk '!a[$0]++' <f
q
w
e
r
$ 

The original code I have found on the internets read:

awk '!_[$0]++'

This was even more perplexing to me as I took _ to have a special meaning in awk, like in Perl, but it turned out to be just a name of an array.

Now, I understand the logic behind the one-liner: each input line is used as a key in a hash array, thus, upon completion, the hash contains unique lines in the order of arrival.

What I would like to learn is how exactly this notation is interpreted by awk. E.g. what the bang sign (!) means and the other elements of this code snippet.

How does it work?

  • title is misleading, it should be $0 (Zero), not $o (o). – Archemar Oct 6 '14 at 21:06
  • 2
    As it's a hash, it's unordered, so "in the order of arrival" isn't actually correct. – Kevin Oct 7 '14 at 6:10
35

Let's see,

 !a[$0]++

first

 a[$0]

we look at the value of a[$0] (array a with whole input line ($0) as key).

If it does not exist ( ! is negation in test will eval to true)

 !a[$0]

we print the input line $0 (default action).

Also, we add one ( ++ ) to a[$0], so next time !a[$0] will evaluate to false.

Nice, find!! You should have a look at code golf!

  • 1
    So the essence is this: the expression in the single quotes is used by awk as a test for each input line; every time the test succeeds awk executes the action in curly braces, which, when omitted is {print}. Thanks! – Alexander Shcheblikin Oct 7 '14 at 1:21
  • 3
    @Archemar: This answer is wrong, see mine. – cuonglm Oct 7 '14 at 3:29
  • @AlexanderShcheblikin in awk, the default action is {print $0}. This means that anything evaluated as true will execute this as default. So for example awk '1' file prints all the lines, awk '$1' file prints all those lines whose first field is not empty or 0, etc. – fedorqui Oct 7 '14 at 10:31
  • 6
    @Gnouc I don't see any serious error in this answer. If that is what you're referring to, the incrementation is indeed applied after the value of the expression is calculated. It's true that the incrementation happens before the printing, but that's a minor imprecision which doesn't affect the basic explanation. – Gilles Oct 7 '14 at 17:23
  • 1
    I found the best explanation for a newbie to understand here in quora: qr.ae/TUIVxM – GP92 Aug 6 '18 at 7:32
29

Here is the processing:

  • a[$0]: look at the value of key $0, in associative array a. If it does not exist, create it.

  • a[$0]++: increment the value of a[$0], return the old value as value of expression. If a[$0] does not exist, return 0 and increment a[$0] to 1 (++ operator returns numeric value).

  • !a[$0]++: negate the value of expression. If a[$0]++ return 0, the whole expression is evaluated to true, make awk performed default action print $0. Otherwise, the whole expression is evaluated to false, causes awk do nothing.

References:

With gawk, we can use dgawk (or awk --debug with newer version) to debug a gawk script. First, create a gawk script, named test.awk:

BEGIN {                                                                         
    a = 0;                                                                      
    !a++;                                                                       
}

Then run:

dgawk -f test.awk

or:

gawk --debug -f test.awk

In debugger console:

$ dgawk -f test.awk
dgawk> trace on
dgawk> watch a
Watchpoint 1: a
dgawk> run
Starting program: 
[     1:0x7fe59154cfe0] Op_rule             : [in_rule = BEGIN] [source_file = test.awk]
[     2:0x7fe59154bf80] Op_push_i           : 0 [PERM|NUMCUR|NUMBER]
[     2:0x7fe59154bf20] Op_store_var        : a [do_reference = FALSE]
[     3:0x7fe59154bf60] Op_push_lhs         : a [do_reference = TRUE]
Stopping in BEGIN ...
Watchpoint 1: a
  Old value: untyped variable
  New value: 0
main() at `test.awk':3
3           !a++;
dgawk> step
[     3:0x7fe59154bfc0] Op_postincrement    : 
[     3:0x7fe59154bf40] Op_not              : 
Watchpoint 1: a
  Old value: 0
  New value: 1
main() at `test.awk':3
3           !a++;
dgawk>

You can see, Op_postincrement was executed before Op_not.

You can also use si or stepi instead of s or step to see more clearly:

dgawk> si
[     3:0x7ff061ac1fc0] Op_postincrement    : 
3           !a++;
dgawk> si
[     3:0x7ff061ac1f40] Op_not              : 
Watchpoint 1: a
  Old value: 0
  New value: 1
main() at `test.awk':3
3           !a++;
  • 3
    @Archemar: Your answer indicate that ! is applied before ++. – cuonglm Oct 7 '14 at 5:37
  • 6
    This answer is wrong. The incrementation happens after the result of the ! operator is calculated. You are confusing operator precedence (!a[$0]++ is parsed like !(a[$0]++)) with order of evaluation (the assignment of the new value of a[$0] happens after the value of the expression has been calculated). – Gilles Oct 7 '14 at 17:21
  • 5
    @Gnouc It says right in the passage you quoted, and if it worked the way you described, this code wouldn't have the desired effect. First the value !x is calculated, where x is the old value of a[$0]. Then a[$0] is set to 1+x. – Gilles Oct 7 '14 at 17:26
  • 7
    I believe that your analysis of what awk does is correct. Sorry if I implied otherwise yesterday. However, your critique of Archemar's answer is wrong. Archemar does not misunderstand precedence, you do, you're confusing precedence with order of evaluation (see my previous comment). If you remove any mention of Archemar's answer in yours, your answer should be correct. As it is, it is focused on proving Archemar wrong, and this is not the case. – Gilles Oct 8 '14 at 7:59
  • 5
    well, at least now I know about awk's debugger ... – Archemar Oct 8 '14 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.