Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Below awk command removes all duplicate lines as explained here:

awk '!seen[$0]++'

If the text contains empty lines, all but one empty line will be deleted.

How can I keep all empty lines whilst deleting all non-empty duplicate lines, using only awk? Please, also include a brief explanation.

share|improve this question
up vote 17 down vote accepted

Another option is to check NF, eg:

awk '!NF || !seen[$0]++'
share|improve this answer


awk '!/./ || !seen[$0]++' file

The main trick is the same, seen[$0]++ creates an entry in the seen associative array whose key is the current line ($0). Therefore, !seen[$0]++ will be false if this line has already been seen. The /./ is checking whether the line contains any non-blank characters, so !/./ matches non blank lines. Combined with || !seen[$0]++ it will ignore all duplicate lines except blank ones and print the rest.

share|improve this answer
awk '/^[[:blank:]]*$/ { print; next; }; !seen[$0]++'

All you have to do is check for an empty (really empty or just blank) line first.

share|improve this answer

Here is another awk solution, similar to @Thor's answer, less concise but more efficient:

awk '!NF {print;next}; !($0 in a) {a[$0];print}' file

With this, we only check a[$0] has existed or not. If not, initializing it then print. In this case, we don't have any reference, assignment to a[$0] if it existed.

share|improve this answer
I did not measure any significant time difference with my 288-line test file. However, your code certainly catches the prize for being the most readable. – Serge Stroobandt Nov 5 '15 at 20:24

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.