How to remove duplicate lines that begin with a pattern and the next line after that?

Question

I want to remove duplicate lines that begin with > and the next line after that.

For example:

>1
ACCGGTTTCCTTGAAATT
>2 
AACCTTCCGGTTAATT
>3 
AACCTTCCGGTTAATT
>1 
ACCGGTTTCCTTGAAATT

As you can see I have the next two duplicated lines:

AACCTTCCGGTTAATT and >1 

However I only want to remove >1 and the next line, so I want and output like:

>1
ACCGGTTTCCTTGAAATT
>2
AACCTTCCGGTTAATT
>3
AACCTTCCGGTTAATT

If I use something like:

awk '!seen[$0]++'  filename

The output is:

>1
ACCGGTTTCCTTGAAATT
>2
AACCTTCCGGTTAATT
>3

Because it removes all duplicated lines and I only want to remove duplicated lines that begin with > and the next line after that.

My true file is about several thousand of lines so I could have several names after the symbol > that could be repeated.

Any suggestions?

Answer 1

You can use getline in your awk to fetch the next line:

awk '/^>/{ if(!seen[$0]++){ print;getline;print } else { getline } }'

---

There is a simpler answer that also handles multiple lines:

awk '/^>/{ skip = seen[$0]++ }
     { if(!skip)print }'

Answer 2

With POSIX tools chest:

paste - - <file | awk '{$1=$1};!seen[$0]++' | tr '\t' '\n'

Answer 3

With awk:

awk 'NR%2==1{l=$0;next} !seen[l"\n"$0]++{print l"\n"$0}' file

• NR%2==1 is true every 2nd line, so the lines with >1, >2 and >3. In this case save that content to a variable l and continue with the next line.
• !seen[l"\n"$0]++ here we don't check for unique lines, we check for 2 unique consecutive lines.
  • If they are unique, print the last line l and the current line $0 with a newline \n between them.

---

The output:

>1 
ACCGGTTTCCTTGAAATT
>2 
AACCTTCCGGTTAATT
>3 
AACCTTCCGGTTAATT