Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Below awk command removes all duplicate lines as explained here:

awk '!seen[$0]++'

If the text contains empty lines, all but one empty line will be deleted.

How can I keep all empty lines whilst deleting all non-empty duplicate lines, using only awk? Please, also include a brief explanation.

share|improve this question

4 Answers 4

up vote 10 down vote accepted

Another option is to check NF, eg:

awk '!NF || !seen[$0]++'
share|improve this answer

Alternatively

awk '!/./ || !seen[$0]++' file

The main trick is the same, seen[$0]++ creates an entry in the seen associative array whose key is the current line ($0). Therefore, !seen[$0]++ will be false if this line has already been seen. The /./ is checking whether the line contains any non-blank characters, so !/./ matches non blank lines. Combined with || !seen[$0]++ it will ignore all duplicate lines except blank ones and print the rest.

share|improve this answer
awk '/^[[:blank:]]*$/ { print; next; }; !seen[$0]++'

All you have to do is check for an empty (really empty or just blank) line first.

share|improve this answer

Here is another awk solution, similar to @Thor's answer, less concise but more efficient:

awk '!NF {print;next} !($0 in a) {a[$0];print}' file

With this, we only check a[$0] has existed or not. If not, initializing it then print. In this case, we don't have any reference, assignment to a[$0] if it existed.

share|improve this answer
    
I did not measure any significant time difference with my 288-line test file. However, your code would certainly catches the prize for being most readable. –  Serge Stroobandt May 21 at 8:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.