ah the ubiquitous but also ominous awk duplicate remover
awk '!a[$0]++'
this sweet baby is the love child of awk's power and terseness. the pinacle of awk one liners. short but powerful and arcane all at once. removes duplicates while maintaining order. a feat unachieved by uniq
or sort -u
which removes only adjacent duplicates or has to break order to remove duplicates.
here is my attempt to explain how this awk one liner works. i took effort in explaining things so that someone who does not know any awk can still follow along. i hope i was able to do so.
first some pseudo code. what this one liner does is basically the following:
for every line:
if i have not seen this line before:
print line
take note that i have now seen this line
i hope you can see how this removes duplicates while maintaining order.
but how does a loop, an if, a print, and a mechanism for storing and retrieving strings fit in 8 characters of awk code? the answer is implicit.
the loop is implicit.
one of the core design philosophies of awk is that it does an implicit loop over every line of input. most code that you write in awk is inside this implicit loop.
the print is implicit.
this
awk '!a[$0]++'
is equivalent to this
awk '!a[$0]++ { print $0 }'
$0
is the awk variable for the current line. print $0
means to print the current line.
the if is implict.
this thing !a[$0]++ { print $0 }
is an awk "rule". a rule consists of a condition and a code block. this !a[$0]++
is the condition and this { print $0 }
is the code block.
a typical awk program consists of one or more rules. for every input line awk tests the condition and if it is true it will execute the code block. if the condition is missing then it is implicit true. if the code block is missing then it is implicit { print $0 }
.
so this thing !a[$0]++
is the conditional. based on which it is decided whether to print the line or not. and it somehow evaluates to true or false based on if the line is a duplicate or not? how does that work?
lets look at the pseudo code again
for every line: # implicit by awk
if i have not seen this line before: # at least we know the conditional part
print line # implicit by awk
take note that i have now seen this line # ???
we understand the loop, the print, and the if. but how does it work so that it evaluates to false only at duplicate lines? and how does it take note of lines already seen?
let's take apart this beast: !a[$0]++
if you know c or java you should already know some of the symbols. the semantics are identical or at least similar.
the exclamation mark (!
) is a negator. it evaluates the expression to a boolean and whatever the result it is negated. if the expression evaluates to true the end result is false and vice versa.
a[..]
is an array. an associative array. other languages name it map or dictionary. in awk all arrays are associative arrays. the a
has no special meaning. it is just a name for the array. it could just as well be x
or eliminatetheduplicate
.
$0
is the current line from the input. this is an awk specific variable.
the plus plus (++
) is a post increment operator. this operator is a bit tricky because it does two things: the value in the variable is incremented. but it also "returns" a value for further processing. the value "returned" is the original, not incremented, value.
! a[ $0 ] ++
negator array current line post increment
how do they work together?
roughly in this order:
$0
is the current line
a[$0]
is the value in the array for the current line
- the post increment (
++
) gets the value from a[$0]
; increments and stores it back to a[$0]
; then "returns" the original value to the next operator in line: the negator.
- the negator (
!
) gets a value from the ++
which was the original value from a[$0]
; it is evaluated to a boolean then negated then passed to the implicit if.
- the if then decides whether to print the line or not.
so that means whether the line gets printed or not, or in the context of this awk program: whether the line is a duplicate or not, is ultimately decided by the value in a[$0]
.
by extension: the mechanism that is taking note whether this line has already been seen must then happen when ++
stores the incremented value back to a[$0]
.
lets look at the pseudo code again
for every line:
if i have not seen this line before: # read from a[$0]
print line
take note that i have now seen this line # write to a[$0]
remember that this operator ++
does two things. increment the value in the variable and return the original value for further processing.
lets try to take the ++
apart. lets take the entire thing apart.
we start with
result = !a[$0]++
take the negator out
tmp = a[$0]++
result = !tmp
now we take the ++
apart. since it does two things we make two lines out of it.
tmp = a[$0] # save the original value to use later
a[$0] = tmp + 1 # increment and store in variable
result = !tmp # continue working with original value
now lets try to put that back in the pseudo code
for every line:
tmp = a[$0] # query if have seen line
a[$0] = tmp + 1 # take note that has seen line
if !tmp: # decide whether to print line or not
print line
so there we have it. we have the loop, the if, the print, the query and the take note. just in a different order than the previous pseudo code.
condensed to 8 characters
!a[$0]++
possible because of awks implicit loop, implicit if, implicit print, and because the ++
does the query and take note at the same time.
remains one question. what is the value of a[$0]
for the first line? or for any line that has not been seen before? the answer is again implicit.
in awk any variable that is used for the first time is implicitly declared and initialized to an empty string. except arrays. arrays are declared and initialized to an empty array.
so when awk reaches this part: a[$0]
it will first create an empty array. then it will initialize the item at $0
with an empty string.
so the value of a[$0]
for the first line is the empty string. same for any following lines that are seen for the first time.
the ++
is a number operator. if given a string it is converted to a number. the empty string converts to zero. any other string would be interpreted as number by some best effort algorithm. zero if string is not a number.
the !
is a boolean operator. if given a number or a string it is converted to boolean. the number zero is false. the empty string is false. anything else is true.
so that means when a line is seen for the first time then a[$0]
is not set. awk does the implicit and it is set to the empty string. the empty string is converted to zero because of ++
then to false because of !
. the result from !
is true so the line gets printed.
the value in a[$0]
is now the number 1.
if a line is seen the second time then a[$0]
is the number 1 which is true and the result from !
is false so it is not printed.
any further encounter of the same line increases the number. since all numbers except zero are true the result from !
will always be false so the line never gets printed again.
that is how the duplicates are detected.
TL;DR: it counts how often a line has been seen. if zero then print. if any other number then no print. it can be short because of many implicits.
bonus: some variants of the one liner and super short explanation of what it does.
replace $0
(entire line) with $2
(second column) will remove duplicates but only based on the second column
$ cat input
x y z
p q r
a y b
$ awk '!a[$2]++' input
x y z
p q r
replace !
(negator) with ==1
(equal to one) will only print lines that are duplicates
$ cat input
a
b
c
b
c
c
$ awk 'a[$0]++==1' input
b
c
replace with >0
(greater than zero) and add {print NR":"$0}
will print all duplicate lines (other than the original) with line number. NR
is a special awk variable containing the line number (record number to be exact).
$ awk 'a[$0]++>0 {print NR":"$0}' input
4:b
5:c
6:c
to print all duplicate including the original will require much more changes because you have to keep track of past line numbers.