1

I want to remove four lines of a fastq file. For example, usually the file looks like this: (for each sample correspond four lines)

@M04241:303:000000000-BR896:1:1102:21438:12389 1:N:0:TATGGCAC
TGTCAGCCGCCGCGGTAATACGGAGGGTCCGAGCGTTATCCGGAATTATTGGGTTTAAAGGGTCCGCAGGCGGGCTTATAAGTCAGGGGTGGAATGGTGCGGCTCAACCGTAGCACTGCCCTTGATACTGTTAGTCTTGAGTTATGGTGGAGTGGCCGGAATATGTAGTGTAGCGGTGAAATGCATAGATATTACATAGAACACCGATCGCGAAGGCAGGTCACTAACCATTTGACTGACGCTGATGGACGAAAGCGTGGGGAGCGAACAGGATTAGATACCCTGGTAGTCCACGCCGGAAACGATGGATACTAGCTGTCGGGCACTTGTGCTCGGCGGCCAAGCGAAAGTGATAAGTATCCCACCTGGGGAGTACGTGCGCAAGAATGAAACTCAAATGAATTGACGG
+
EGGGGGGGGGGGGGGGGGGGGGGGDE@FFGEEEGGGGDGFEFGGGGGGGGGGGGGGGGGGGGGGGDGEFFGGGCGGFDF<DGGFGGGGGGGG7FFG?FDF:FGGGFCGGGGFEGGGF:>GGGG>F>DE@GG6@GGG@G9<EGGGG9FGGGGGG7FGGDDEFGGGGGGGGGGGGGGGGCEFGGGGFG?EFFCFGGGGGGFGG?GGGGGGGG=EGEGGGGGGGGGGGFGCGGFGGGGCFFF6CD7DDFFFFFED9:BFCBEE@DEF:@EGCFCF@FFFD?=A:CFEF0<C<A>FB>@6+C,@GFFGFDGGF<AFEFB+FEECGFF9FDFAC6@+:@FC:GFC,CFC,EFGE,9FFCGFF<@;6:,FD,D:FGGFFGF7@8+7,,CF<<6CF<CC-CA@<GEGFE@6@A,CB
@M04241:303:000000000-BR896:1:1103:11464:7575 1:N:0:TATGGCAC
GTCAATTTCTTTGCGTTTCAATCTTGCGATCGTACTCCCCAGGTGGGATACTTATCACTTTCGCTTAGTCACTGAGATAAATCCCAACAACTAGTGTCCATCGTTTACGGCGTGGACTACCAGGGTATCTAATCCTGTTCGCTCCCCACGCTTTCGTCCATCAGCGTCAGTATATGGTTAGTGACCTGCCTTCGCGATCGGTGTTCTATGTAATATCTATGCATTTCACCGCTACACTACATATTCCGGCCACTCCACCATAACTCAAGACTAACAGTATCAAAGGCAGTGCTACGGTTGAGCCGCACCATTTCACCCCTGACTTATCAGCCCGCCTGCGGACCCTTTAAACCCAATAATTCCGGATAACGCTCGGACCCTCCGTATTACCGCGGCTGCTGGC
+
CCCCCGGGGGGGG-FCFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGFFGGDFFGFGFGGGGGGGGGGGGGGGGGGGGGGGGGEGGEGGGGDGGG4FFGGGGGGGGGGGGGGGGGGGGGEGGGGGGFGGGFFGGGGGGGGGGGGGGGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFFFGFFFGFGGGGGGGGGGGGGGGGGGGFGGFFGGGGGGGGGGGGGGGGGGGCDGGGGGGGGFCFGGGFGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGGGGGGGGGFGGGGGCGEFFGGEGGGGGGGGGGGGGGGGGDGGGGFFCGGGGGGGGGGGGFGGGDGGGGGGGGGGGGFGGGGGGGGGGGGGGGGG
@M04241:303:000000000-BR896:1:1103:23291:21403 1:N:0:TATGGCAC
CTGCGGCACCGCAGGGCAAGCCCCCCGACGCCTAGCCCACATCGTTTAGGGCGTGGACTACCAGGGTATCTAATCCTGTTTGCTCCCCACGCTTTCGCGCCTCAGCGTCAGTGCCGGACCAGAGAGCCGCTTTCGCCACCGGTGTTCCACCCAATATCTACGAATTTCACCTCTACACTGGGTATTCCACCCTCCTCTTCCGGACTCGAGCACCGCAGTCTCGGCTGCACCTCCGGGGTTGAGCCCCGGGCTTTCACAGCCGACTTGCGACGCCGCCTACGCGCCCTTTACGCCCAGTGATTCCGAACAACGCTAGCACCCTCCGTCTTACCGCGGCGGCTGAC
+
CCCCCGGGGGG>FGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@@FGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

But I found one sample that have empty two of the four lines empty like this:

@M04241:303:000000000-BR896:1:1103:11464:7575 1:N:0:TATGGCAC

+

@M04241:303:000000000-BR896:1:1103:23291:21403 1:N:0:TATGGCAC
CTGCGGCACCGCAGGGCAAGCCCCCCGACGCCTAGCCCACATCGTTTAGGGCGTGGACTACCAGGGTATCTAATCCTGTTTGCTCCCCACGCTTTCGCGCCTCAGCGTCAGTGCCGGACCAGAGAGCCGCTTTCGCCACCGGTGTTCCACCCAATATCTACGAATTTCACCTCTACACTGGGTATTCCACCCTCCTCTTCCGGACTCGAGCACCGCAGTCTCGGCTGCACCTCCGGGGTTGAGCCCCGGGCTTTCACAGCCGACTTGCGACGCCGCCTACGCGCCCTTTACGCCCAGTGATTCCGAACAACGCTAGCACCCTCCGTCTTACCGCGGCGGCTGAC
+
CCCCCGGGGGG>FGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@@FGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
@M04241:303:000000000-BR896:1:1103:26180:21941 1:N:0:TATGGCAC
CCGCCAATTTCTTTGAGTTTCAGCCTTGCGACCATACTCCCCAGGCGGGGTACTTAACACTTTTGATTCGGCAGTGCACCCATGTTAGTCCACTACCTAGTACCCATCGTTTAGGGCTAGGACTACCGGGGTATCTAATCCCGTTCGCTACCCTAGCTTTCGCGCCTCAGCGTCAGAAGAGGTCCAGCACGTCGCTTTCGCCACCGGCGTTCCTTCCGATCTCTACGCATTTCACCGCTCCACCGGAAGTTCCACATGCCCCTACCTCCCTCGAGATTGGCAGTTTCGAAGGCAGTTCTACAGTTGAGCTGCAGGATTTCACCTCCGACTGACCTATCCGCCTACGCGCCCTTTAAGCCCAGTGATTCCGAACAACGTTCGC
+
CCCCCGEGGGGGGGGGGEGGGGGGGGGGDFGGGGGGGGGGGGGEGGGGGGEFGGGFFFFGGGGGG,CEFGGGGGGGGGG?GGGGGG9FFGGGGGGGCGGGGGGGGGCFGGGG@GGGGGFGGGGGGGGGCGGFGGGGGGGGGGGGGGGGGGGGGGGFFGGGGGGGGGDEGGGGGGGDGGGGFGFFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGEGGGDGEFGGEEGGGGFGGGGGGGGGGGGGGGGGGGGGEF?GGGEGGEEFEFFDFFGFGGFGGGGGGFFFGFGGGGGGGGGFGGGGFCGGGGGGGGGFFGGGGGGGGGGGGGGGFF@7GGGGGGGGGGGGGGGFDFCGGGGFEFGGFGGGGGGGGFGFEGGGG
@M04241:303:000000000-BR896:1:1102:21438:12389 1:N:0:TATGGCAC
TGTCAGCCGCCGCGGTAATACGGAGGGTCCGAGCGTTATCCGGAATTATTGGGTTTAAAGGGTCCGCAGGCGGGCTTATAAGTCAGGGGTGGAATGGTGCGGCTCAACCGTAGCACTGCCCTTGATACTGTTAGTCTTGAGTTATGGTGGAGTGGCCGGAATATGTAGTGTAGCGGTGAAATGCATAGATATTACATAGAACACCGATCGCGAAGGCAGGTCACTAACCATTTGACTGACGCTGATGGACGAAAGCGTGGGGAGCGAACAGGATTAGATACCCTGGTAGTCCACGCCGGAAACGATGGATACTAGCTGTCGGGCACTTGTGCTCGGCGGCCAAGCGAAAGTGATAAGTATCCCACCTGGGGAGTACGTGCGCAAGAATGAAACTCAAATGAATTGACGG
+
EGGGGGGGGGGGGGGGGGGGGGGGDE@FFGEEEGGGGDGFEFGGGGGGGGGGGGGGGGGGGGGGGDGEFFGGGCGGFDF<DGGFGGGGGGGG7FFG?FDF:FGGGFCGGGGFEGGGF:>GGGG>F>DE@GG6@GGG@G9<EGGGG9FGGGGGG7FGGDDEFGGGGGGGGGGGGGGGGCEFGGGGFG?EFFCFGGGGGGFGG?GGGGGGGG=EGEGGGGGGGGGGGFGCGGFGGGGCFFF6CD7DDFFFFFED9:BFCBEE@DEF:@EGCFCF@FFFD?=A:CFEF0<C<A>FB>@6+C,@GFFGFDGGF<AFEFB+FEECGFF9FDFAC6@+:@FC:GFC,CFC,EFGE,9FFCGFF<@;6:,FD,D:FGGFFGF7@8+7,,CF<<6CF<CC-CA@<GEGFE@6@A,CB

How can I detect this empty lines and remove from a fastq file? I know the number of the line, however it is a huge file I cannot open normally so I need a command to detect that the two lines are empty and to remove the four lines associated to that sample.

Thanks!!

  • 1
    Please remember when editing your question that most of us here are not bioinformatics people. Try not to use terms like "fasta" or "sequence", and instead phrase your question in computational terms. – roaima Jun 27 '18 at 14:18
  • It's a fastq file, as in the title, not a fasta. en.wikipedia.org/wiki/FASTQ_format – Gerard H. Pille Jun 27 '18 at 14:53
  • @GerardH.Pille I've heard of fasta but not of fastq. Neither word really means anything to me except to tag as a bioinformatics question. – roaima Jun 27 '18 at 22:22
  • @roaima, I'd never heard of any of them, thought to correct a typo in the title, but it didn't take two minutes on duckduckgo. – Gerard H. Pille Jun 28 '18 at 1:52
2
sed 'N;N;N;/\n\n/d' file.fastq >new-file.fastq

This would read in the four lines of a FastQ record and then check whether it contains two consecutive newline characters. If it does, the whole record is ignored. If it doesn't, it is printed. This would repeat for all entries in the file. All printed records goes to a new file (here new-file.fastq).

The sed script, with annotations:

         # (implicit: read a line)
N;       # read a second line, append it to the pattern space with embedded \n in-between
N;       # read a third line
N;       # read a fourth line
/\n\n/d  # if there are two consecutive newlines, delete and continue from top
         # (implicit: print)

Comment from a colleague:

Fastq records are often paired, and software tends to throw a fit when pair mates are not found without explicitly telling it mates are missing. Several tools have a minimum length option like trimmomatic that will keep pairing, and separate orphaned records.

This means that if the reads in your file are paired, and one of the pairs is empty, you would mess up the pairing by just removing the empty record.

Removing the mate of the empty read would be a lot more complicated, unless using already existing bioinformatics tools. With the tools in the standard Unix tool chest, it would probably involve saving the empty reads to a separate file, and then, by using their FastQ headers, scan for and delete the corresponding mates.

The data shown in the question seems to only be unpaired reads though.

  • the reads are already paired so it is not need to have a mate. – edx Jun 28 '18 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.