Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have 2 files. File 1, has an identifier (eg. D7MHBF:11:1449:1988) and every new entry starts with @. It has few more fields, which are not important in our analysis.

File 2 consists of a column of Identifiers. If the Identifier of File 2 is exactly matching with the Identifier of File 1 then, delete the whole three-line entry in File 1.(P.S. Every entry starts with @). File 1:

@D7MHBF:11:1449:1988  1:N
NGCCCATTCTATGCAACAAATTCAACAAAGTCTCCCA
+
#1=DDFFDFHHDHIGGFIIIIIIIHHIHIGDGIGGEH
@D7MHBF:11:1681:1959  1:N
NGGTTTAGAGCCTGTATTTTGAACGTAATAGCAGGGT
+
#1=ADDFFHHDDHEHIHJJJICCFHGEIGJGGGIJJF
@D7MHBF:11:1778:1946  1:N
NTGTGGGTTCAACAATGGGCTCATTGCCATCAGCGAG
+
#1=ADDFDHHGHFHGGIIJJJGIIJIFIIJJJIJIIJ
@D7MHBF:11:1854:1955  1:N
NTTGATTGTGCTTCTTATGAGCAAAAATGTTGATGAC
+
#1=DFFFFHHHGHHIJEHIIGGGGIJJJJIHIIHIII
@D7MHBF:11:1903:1972  1:N
NACTACTATACTATGAAGTGGCTATGACAATTGTCTT
+
#1=DFFFFHHHGHHIJEHIIIGGGGIJJJJIHIIHII

File 2:

D7MHBF:11:1449:1988
D7MHBF:11:13844:51576
D7MHBF:11:13838:51641
D7MHBF:11:1778:1946
D7MHBF:11:1903:1972                                                                                                                                                                                                                                                                                                                                                                                                    

Expected Output:

@D7MHBF:11:1681:1959  1:N
NGGTTTAGAGCCTGTATTTTGAACGTAATAGCAGGGT
+
#1=ADDFFHHDDHEHIHJJJICCFHGEIGJGGGIJJF
@D7MHBF:11:1854:1955  1:N
NTTGATTGTGCTTCTTATGAGCAAAAATGTTGATGAC
+
#1=DFFFFHHHGHHIJEHIIGGGGIJJJJIHIIHIII

I am new in Bioinformatics and Programming.

share|improve this question
    
In your example not every line starts with an @, this does not confirm with your description, please correct one or the other. –  Anthon Feb 4 at 16:06
    
Hi Anthon, Every line starts with @. This is the format of the Fastq File. One complete line consists of 4 lines. –  Namrata Feb 5 at 12:52
    
@Anthon by line, the OP means entry. –  terdon Feb 6 at 19:16
    
@terdon Yes that is now clear, thanks. (I had not seen the OP comment while not having been addressed with @). We are fortunate the OP doesn't use line for what I would call a file as well. –  Anthon Feb 6 at 19:44

3 Answers 3

up vote 0 down vote accepted

Here's one solution (copy/paste into your terminal):

perl -pe 's/\n/xxx/g' file1 | sed 's/xxx@/\n@/g; s/xxx$/\n/' |
 grep -vFf file2 | sed 's/xxx/\n/g'

Explanation

There are three separate commands there, each of which pipes (|) its output to the next. The first two just get each record on a separate line by replacing \n with xxx (the xxx is just so we can know where the line breaks are). If you run them alone they do:

$ perl -pe 's/\n/xxx/g' file1 | sed 's/xxx@/\n@/g; s/xxx$/\n/'
@D7MHBF:11:1449:1988  1:NxxxNGCCCATTCTATGCAACAAATTCAACAAAGTCTCCCAxxx+xxx#1=DDFFDFHHDHIGGFIIIIIIIHHIHIGDGIGGEH
@D7MHBF:11:1681:1959  1:NxxxNGGTTTAGAGCCTGTATTTTGAACGTAATAGCAGGGTxxx+xxx#1=ADDFFHHDDHEHIHJJJICCFHGEIGJGGGIJJF
@D7MHBF:11:1778:1946  1:NxxxNTGTGGGTTCAACAATGGGCTCATTGCCATCAGCGAGxxx+xxx#1=ADDFDHHGHFHGGIIJJJGIIJIFIIJJJIJIIJ
@D7MHBF:11:1854:1955  1:NxxxNTTGATTGTGCTTCTTATGAGCAAAAATGTTGATGACxxx+xxx#1=DFFFFHHHGHHIJEHIIGGGGIJJJJIHIIHIII
@D7MHBF:11:1903:1972  1:NxxxNACTACTATACTATGAAGTGGCTATGACAATTGTCTTxxx+xxx#1=DFFFFHHHGHHIJEHIIIGGGGIJJJJIHIIHII

As you can see, now each entry is on its own line. So, you can now pass this to grep and tell it to print all lines that do not (-v) match any of the patterns found in file2 (-f file2). This prints:

$ perl -pe 's/\n/xxx/g' file1 | sed 's/xxx@/\n@/g; s/xxx$/\n/' | grep -vFf file2  
@D7MHBF:11:1681:1959  1:NxxxNGGTTTAGAGCCTGTATTTTGAACGTAATAGCAGGGTxxx+xxx#1=ADDFFHHDDHEHIHJJJICCFHGEIGJGGGIJJF
@D7MHBF:11:1854:1955  1:NxxxNTTGATTGTGCTTCTTATGAGCAAAAATGTTGATGACxxx+xxx#1=DFFFFHHHGHHIJEHIIGGGGIJJJJIHIIHIII

Finally, we run it through sed again, to replace xxx with newlines and get the desired output:

$ perl -pe 's/\n/xxx/g' file1 | sed 's/xxx@/\n@/g; s/xxx$/\n/' | 
    grep -vFf file2  | sed 's/xxx/\n/g' 
@D7MHBF:11:1681:1959  1:N
NGGTTTAGAGCCTGTATTTTGAACGTAATAGCAGGGT
+
#1=ADDFFHHDDHEHIHJJJICCFHGEIGJGGGIJJF
@D7MHBF:11:1854:1955  1:N
NTTGATTGTGCTTCTTATGAGCAAAAATGTTGATGAC
+
#1=DFFFFHHHGHHIJEHIIGGGGIJJJJIHIIHIII
share|improve this answer
    
Thank you. It works. –  Srilakshmi Feb 19 at 13:17

This problem can be solved with the pattern matching and text processing features of the language TXR.

Note that @@ encodes a literal @ character to be matched.

@(next "file2.txt")
@(bind idhash @(hash :equal-based))
@(repeat)
@id
@(do (set [idhash id] t))
@(end)
@(next "file1.txt")
@(bind outstream @(open-file "file3.txt" "w"))
@(repeat)
@@@ident@\ @rest
@  (collect)
@line
@  (until)
@@@(skip)
@  (end)
@  (require (not (gethash idhash ident)))
@  (output :continue outstream)
@@@ident @rest
@    (repeat)
@line
@    (end)
@  (end)
@(end)

Run:

$ txr del.txr 
$ cat file3.txt 
@D7MHBF:11:1681:1959  1:N
NGGTTTAGAGCCTGTATTTTGAACGTAATAGCAGGGT
+
#1=ADDFFHHDDHEHIHJJJICCFHGEIGJGGGIJJF
@D7MHBF:11:1854:1955  1:N
NTTGATTGTGCTTCTTATGAGCAAAAATGTTGATGAC
+
#1=DFFFFHHHGHHIJEHIIGGGGIJJJJIHIIHIII
share|improve this answer
    
Hi.. It says txr: command not found –  Srilakshmi Feb 27 at 14:22
    
Executables of txr here. OSX 10.7.3 included: nongnu.org/txr/#downloads –  Kaz Feb 28 at 16:52

Awk solution:

BEGIN {
  while (getline < "file2.txt") {
    idhash[$0]
  }
  RS="@"
  FS=" "
}

{ if ($1 in idhash) printf("%s%s", NR == 1 ? "" : "@", $0); }

Run:

$ awk -f del.awk file1.txt
@D7MHBF:11:1681:1959  1:N
NGGTTTAGAGCCTGTATTTTGAACGTAATAGCAGGGT
+
#1=ADDFFHHDDHEHIHJJJICCFHGEIGJGGGIJJF
@D7MHBF:11:1854:1955  1:N
NTTGATTGTGCTTCTTATGAGCAAAAATGTTGATGAC
+
#1=DFFFFHHHGHHIJEHIIGGGGIJJJJIHIIHIII
share|improve this answer
    
Hi Kaz, It is not giving me the right output. I am getting Output: D7MHBF:11:1449:1988 1:N NGCCCATTCTATGCAACAAATTCAACAAAGTCTCCCA + #1=DDFFDFHHDHIGGFIIIIIIIHHIHIGDGIGGEH D7MHBF:11:1778:1946 1:N NTGTGGGTTCAACAATGGGCTCATTGCCATCAGCGAG + #1=ADDFDHHGHFHGGIIJJJGIIJIFIIJJJIJIIJ –  Srilakshmi Feb 27 at 14:13
    
It is giving wrong ouput. –  Srilakshmi Feb 27 at 14:24
    
Worsk for me with mawk on an Ubuntu system. –  Kaz Feb 27 at 18:39
    
I am using a Mac Terminal. I do not know if that is the problem. Thanks. –  Srilakshmi Feb 28 at 11:01
    
@Srilakshmi I have OSX in a VirtualBox image; I will try it later. –  Kaz Feb 28 at 16:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.