0

I have recently come across awk seen option. I can see it is removing duplicates in files. But unsure how it works tried google but didnt find any results, need help in clarification for the same.

cat tes
1
2
3
1
1
1
3
4

with awk seen output

cat tes | awk '!seen[$0]++'

1
2
3
4
1
  • 1
    seen is not an option or a command. It is merely a user-defined array name. It is called "seen" because it is an associative array of the string values that have already occurred in the input file. – Paul_Pedant Aug 13 '20 at 10:07
3

seen is the arbitrary name of an associative array. It is not an option of any kind. You could use a or b or most other names in its place.

The code !seen[$0]++ consists of a test and an increment.

If seen[$0], i.e. the value of the array element associated with the key $0, the current line of input, is zero (or empty), then the test is true.

The value in the array corresponding to the key $0 is then incremented, which means that the test will be false all other times that the same value of $0 is found.

The effect is that the test is true the first time a particular line is seen in the input, and false all other times.

Whenever the a test with no associated action is true, the default action is triggered. The default action is the equivalent of { print } or { print $0 }, which prints the current record, which for all accounts and purposes in this example is the current unmodified line of input.

2
  • Thanks as always – ajay Aug 13 '20 at 10:11
  • Changing the $0 can extend the capabilities of this expression. For example, indexing by $5 lists lines with at most one instance of field 5, indexing by $2 FS $4 is unique on two non-adjacent fields. – Paul_Pedant Aug 13 '20 at 10:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.