3

I want to remove duplicate lines that begin with > and the next line after that.

For example:

>1
ACCGGTTTCCTTGAAATT
>2 
AACCTTCCGGTTAATT
>3 
AACCTTCCGGTTAATT
>1 
ACCGGTTTCCTTGAAATT

As you can see I have the next two duplicated lines:

AACCTTCCGGTTAATT and >1 

However I only want to remove >1 and the next line, so I want and output like:

>1
ACCGGTTTCCTTGAAATT
>2
AACCTTCCGGTTAATT
>3
AACCTTCCGGTTAATT

If I use something like:

awk '!seen[$0]++'  filename

The output is:

>1
ACCGGTTTCCTTGAAATT
>2
AACCTTCCGGTTAATT
>3

Because it removes all duplicated lines and I only want to remove duplicated lines that begin with > and the next line after that.

My true file is about several thousand of lines so I could have several names after the symbol > that could be repeated.

Any suggestions?

3 Answers 3

3

You can use getline in your awk to fetch the next line:

awk '/^>/{ if(!seen[$0]++){ print;getline;print } else { getline } }'

There is a simpler answer that also handles multiple lines:

awk '/^>/{ skip = seen[$0]++ }
     { if(!skip)print }'
2
  • Thank you very much. This respond exactly my question. I also had cases with more than two lines after >, thus I use: awk '/^>/ {printf("\n%s\n", $0); next; } {printf ("%s", $0);} END {print("\n");}' filename | awk '/^>/{ if(!seen[$0]++){ print;getline;print } The first code to linearize the sequence and your code to eliminate repetitive sequences that begin with >, just if some one else have the same problem Thank you very much again. Commented Sep 23, 2015 at 23:01
  • I added a simpler solution that should also handle this case.
    – meuh
    Commented Sep 24, 2015 at 6:02
2

With POSIX tools chest:

paste - - <file | awk '{$1=$1};!seen[$0]++' | tr '\t' '\n'
3
  • Nice. I'd use paste instead of sed. Commented Sep 23, 2015 at 21:12
  • I don't see the problem: paste - - < file | awk '{$1=$1} !seen[$0]++' | tr '\t' '\n' Commented Sep 24, 2015 at 13:36
  • @glennjackman: In my above comment, I mean trailing spaces. Anyway, your one is better. Added to my answer with posixified awk.
    – cuonglm
    Commented Sep 24, 2015 at 16:13
0

With awk:

awk 'NR%2==1{l=$0;next} !seen[l"\n"$0]++{print l"\n"$0}' file
  • NR%2==1 is true every 2nd line, so the lines with >1, >2 and >3. In this case save that content to a variable l and continue with the next line.
  • !seen[l"\n"$0]++ here we don't check for unique lines, we check for 2 unique consecutive lines.
    • If they are unique, print the last line l and the current line $0 with a newline \n between them.

The output:

>1 
ACCGGTTTCCTTGAAATT
>2 
AACCTTCCGGTTAATT
>3 
AACCTTCCGGTTAATT
11
  • 1) NR%2 is enough (without ==1); 2)[l$0] there is no reason to put \n inside of index 3)* print l;print looks better than print l"\n"$0
    – Costas
    Commented Sep 23, 2015 at 20:08
  • @Costas I used [l"\n"$0] on purpose, because a line could contain l followed by $0 (and wouldn't be printed), but not with a newline between: it would be 2 lines then.
    – chaos
    Commented Sep 23, 2015 at 20:25
  • Line couldn't contain l followed by $0 because each line is $0 only
    – Costas
    Commented Sep 23, 2015 at 20:32
  • @Costas The current line not, but a line that was processed before and is now index of seen.
    – chaos
    Commented Sep 23, 2015 at 20:34
  • Understand. You mean something like >1A AAB and >1 AAAB? Agree. I hope it is not the case, but for future [l,$0] looks better then [l"\n"$0]
    – Costas
    Commented Sep 23, 2015 at 20:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .