25

Below awk command removes all duplicate lines as explained here:

awk '!seen[$0]++'

If the text contains empty lines, all but one empty line will be deleted.

How can I keep all empty lines whilst deleting all non-empty duplicate lines, using only awk? Please, also include a brief explanation.

6 Answers 6

37

Another option is to check NF, eg:

awk '!NF || !seen[$0]++'

Or equivalently:

awk '!(NF && seen[$0]++)'
14

Alternatively

awk '!/./ || !seen[$0]++' file

The main trick is the same, seen[$0]++ creates an entry in the seen associative array whose key is the current line ($0). Therefore, !seen[$0]++ will be false if this line has already been seen. The /./ is checking whether the line contains any non-blank characters, so !/./ matches non blank lines. Combined with || !seen[$0]++ it will ignore all duplicate lines except blank ones and print the rest.

1
  • I think this should've been the accepted answer. +1 for explanation!
    – S.S. Anne
    Mar 26, 2019 at 21:42
7

Here is another awk solution, similar to @Thor's answer, less concise but more efficient:

awk '!NF {print;next}; !($0 in a) {a[$0];print}' file

With this, we only check a[$0] has existed or not. If not, initializing it then print. In this case, we don't have any reference, assignment to a[$0] if it existed.

2
  • I did not measure any significant time difference with my 288-line test file. However, your code certainly catches the prize for being the most readable. Nov 5, 2015 at 20:24
  • Or, shorter '!($0 in a) {if (NF) a[$0]; print}'. - more readably: !($0 in a) {if (NF) {a[$0]}; print}'. By ensuring the $0 is only stored if the line is non-empty, ($0 in a) will always be false for empty lines.
    – AdminBee
    May 24, 2022 at 11:16
5
awk '/^[[:blank:]]*$/ { print; next; }; !seen[$0]++'

All you have to do is check for an empty (really empty or just blank) line first.

0
2

Logical operator NOТ move outside of parentheses:

awk '!(NF && seen[$0]++)'
0
awk '/^$/ || !seen[$0]++' filename

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .