3

I want to remove duplicate lines that begin with > and the next line after that.

For example:

>1
ACCGGTTTCCTTGAAATT
>2 
AACCTTCCGGTTAATT
>3 
AACCTTCCGGTTAATT
>1 
ACCGGTTTCCTTGAAATT

As you can see I have the next two duplicated lines:

AACCTTCCGGTTAATT and >1 

However I only want to remove >1 and the next line, so I want and output like:

>1
ACCGGTTTCCTTGAAATT
>2
AACCTTCCGGTTAATT
>3
AACCTTCCGGTTAATT

If I use something like:

awk '!seen[$0]++'  filename

The output is:

>1
ACCGGTTTCCTTGAAATT
>2
AACCTTCCGGTTAATT
>3

Because it removes all duplicated lines and I only want to remove duplicated lines that begin with > and the next line after that.

My true file is about several thousand of lines so I could have several names after the symbol > that could be repeated.

Any suggestions?

3

You can use getline in your awk to fetch the next line:

awk '/^>/{ if(!seen[$0]++){ print;getline;print } else { getline } }'

There is a simpler answer that also handles multiple lines:

awk '/^>/{ skip = seen[$0]++ }
     { if(!skip)print }'
  • Thank you very much. This respond exactly my question. I also had cases with more than two lines after >, thus I use: awk '/^>/ {printf("\n%s\n", $0); next; } {printf ("%s", $0);} END {print("\n");}' filename | awk '/^>/{ if(!seen[$0]++){ print;getline;print } The first code to linearize the sequence and your code to eliminate repetitive sequences that begin with >, just if some one else have the same problem Thank you very much again. – Eric González Sep 23 '15 at 23:01
  • I added a simpler solution that should also handle this case. – meuh Sep 24 '15 at 6:02
2

With POSIX tools chest:

paste - - <file | awk '{$1=$1};!seen[$0]++' | tr '\t' '\n'
  • Nice. I'd use paste instead of sed. – glenn jackman Sep 23 '15 at 21:12
  • I don't see the problem: paste - - < file | awk '{$1=$1} !seen[$0]++' | tr '\t' '\n' – glenn jackman Sep 24 '15 at 13:36
  • @glennjackman: In my above comment, I mean trailing spaces. Anyway, your one is better. Added to my answer with posixified awk. – cuonglm Sep 24 '15 at 16:13
0

With awk:

awk 'NR%2==1{l=$0;next} !seen[l"\n"$0]++{print l"\n"$0}' file
  • NR%2==1 is true every 2nd line, so the lines with >1, >2 and >3. In this case save that content to a variable l and continue with the next line.
  • !seen[l"\n"$0]++ here we don't check for unique lines, we check for 2 unique consecutive lines.
    • If they are unique, print the last line l and the current line $0 with a newline \n between them.

The output:

>1 
ACCGGTTTCCTTGAAATT
>2 
AACCTTCCGGTTAATT
>3 
AACCTTCCGGTTAATT
  • 1) NR%2 is enough (without ==1); 2)[l$0] there is no reason to put \n inside of index 3)* print l;print looks better than print l"\n"$0 – Costas Sep 23 '15 at 20:08
  • @Costas I used [l"\n"$0] on purpose, because a line could contain l followed by $0 (and wouldn't be printed), but not with a newline between: it would be 2 lines then. – chaos Sep 23 '15 at 20:25
  • Line couldn't contain l followed by $0 because each line is $0 only – Costas Sep 23 '15 at 20:32
  • @Costas The current line not, but a line that was processed before and is now index of seen. – chaos Sep 23 '15 at 20:34
  • Understand. You mean something like >1A AAB and >1 AAAB? Agree. I hope it is not the case, but for future [l,$0] looks better then [l"\n"$0] – Costas Sep 23 '15 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.