3

I have a fastq file with barcode sequences appended at the header line started with @ after the last :. This pattern repeats every four lines. Below is an example:

@FCID:1:1101:15473:1334 1:N:0:TATTTGCGACAA
AGTGGACTAGGGGATGCCAGCCGCCGCGGTAATACGTAGGTGGCAAGCGTTATCCGGATTTATTGGGCGTAAAGGGAACGCAGGCGGTCTTTTAAGTCTGATGTGAAAGCCTTCGGCTTAACCGGAGTAGTGCTTTGGAAACTGTGCAGCTCGAGTGCAGGAGAGGTAAGCGGAATTCCTAGTGTAGCGGTGAAATGCGTAGATATTAGGAGGAACACCAGTGGCGAAGGCGGCTTACTGGACTGTAACT
+
AAAABFFFFFFCGGGGGGGGGGGGGGGGGGGGGHHHHHHGHHGGGHGHGGGGHHHGGGGGHHHHHHHHGGGGHHHGHHGGGGGGGGGGGGHHHHHHHGHGHHHHHHHHFHHHHHHGGGGHHHHGGGGGHHHHHHHHHHGHHHHHHFHHFHGGGGDFHHHHH.EGGGBFFGGGGGGEFFFGGGGFFGGGF-DFEFFFFFFA.-./FFFFBFFFBFFFFFFA?;/B?F@DCFEAAF-@FFBBBBFFEFFFB;
@FCID:1:1101:15528:1336 1:N:0:GCGGGAAAAAAA
GAATTGGACGAGTGCCAGCAGCCGCGGTAATACGTAGGTGGCAAGCGTTATCCGGAATTATTGGGCGTAAAGAGGGAGCAGGCGGCAGCAAAGGTCTGTGGTGAAAGACTGAAGCTTAACTTCAGTAAGCCATAGAAACCGGGCAGCTAGAGTGCAGGAGAGGATCGTGGAATTCCATGTGTAGCGGTGAAATGCGTAGATATATGGAGGAACACCAGTGGCGAAGGCGACGATCTGGCCTGCAACTGAC
+
DDDDDFFFFCDCGGGGGGGGGGHGGGGGGGHHHHHHHGHHGHHHGHGGGGHHHGGGGGHHHHHHHHGGGGHHGHHGGGGHHHGGGGGGGHHHHGGHHHHHHHGHHHHHHHHHHHHGHHHGHGHHHHHHHHHHHHHHHHHHGGGGGGGHHHHHGHGHHHGGHGDHHGDFFGGGGGGGGGGFGGGFGGG9?EGFGGFFAD;EFFFFFFFFFFFFFFFDEEFFFFFFF-DE->CFFEEAFFFFFFFBFFFFF0

My goal is to append the barcodes into the sequence reads every 2nd line and everything else is unchanged. Below is my expected output (the barcodes are the last 12 letters of each sequence line).

@FCID:1:1101:15473:1334 1:N:0:TATTTGCGACAA
AGTGGACTAGGGGATGCCAGCCGCCGCGGTAATACGTAGGTGGCAAGCGTTATCCGGATTTATTGGGCGTAAAGGGAACGCAGGCGGTCTTTTAAGTCTGATGTGAAAGCCTTCGGCTTAACCGGAGTAGTGCTTTGGAAACTGTGCAGCTCGAGTGCAGGAGAGGTAAGCGGAATTCCTAGTGTAGCGGTGAAATGCGTAGATATTAGGAGGAACACCAGTGGCGAAGGCGGCTTACTGGACTGTAACTTATTTGCGACAA
+
AAAABFFFFFFCGGGGGGGGGGGGGGGGGGGGGHHHHHHGHHGGGHGHGGGGHHHGGGGGHHHHHHHHGGGGHHHGHHGGGGGGGGGGGGHHHHHHHGHGHHHHHHHHFHHHHHHGGGGHHHHGGGGGHHHHHHHHHHGHHHHHHFHHFHGGGGDFHHHHH.EGGGBFFGGGGGGEFFFGGGGFFGGGF-DFEFFFFFFA.-./FFFFBFFFBFFFFFFA?;/B?F@DCFEAAF-@FFBBBBFFEFFFB;
@FCID:1:1101:15528:1336 1:N:0:GCGGGAAAAAAA
GAATTGGACGAGTGCCAGCAGCCGCGGTAATACGTAGGTGGCAAGCGTTATCCGGAATTATTGGGCGTAAAGAGGGAGCAGGCGGCAGCAAAGGTCTGTGGTGAAAGACTGAAGCTTAACTTCAGTAAGCCATAGAAACCGGGCAGCTAGAGTGCAGGAGAGGATCGTGGAATTCCATGTGTAGCGGTGAAATGCGTAGATATATGGAGGAACACCAGTGGCGAAGGCGACGATCTGGCCTGCAACTGACGCGGGAAAAAAA
+
DDDDDFFFFCDCGGGGGGGGGGHGGGGGGGHHHHHHHGHHGHHHGHGGGGHHHGGGGGHHHHHHHHGGGGHHGHHGGGGHHHGGGGGGGHHHHGGHHHHHHHGHHHHHHHHHHHHGHHHGHGHHHHHHHHHHHHHHHHHHGGGGGGGHHHHHGHGHHHGGHGDHHGDFFGGGGGGGGGGFGGGFGGG9?EGFGGFFAD;EFFFFFFFFFFFFFFFDEEFFFFFFF-DE->CFFEEAFFFFFFFBFFFFF0

I tried to use awk, but this does not work.

awk '(FNR) % 4 == 1 { -F; seq=$8; next }
     (FNR) % 4 == 2 { line[FNR]=$0; print $0 seq}' R1test.fq > R1test_new.fq

Could anyone help?

2
  • How sure are you that you don't have sequences spanning more than a single line? The fastq format allows that, you can have multiple sequence lines. So relying on lineNumber % 4 is risky. It is fine if you are 100% sure you never have more than 4 lines per record, but you need to be sure. – terdon Jun 14 at 17:39
  • This is indeed a text-processing question at its core, so is 100% on topic here. However, you might be interested in our sister site, Bioinformatics where people who know these formats and the kind of problem you might face hang out. – terdon Jun 14 at 17:52
5

I will make the following assumptions:

  1. All of your records have exactly 4 lines. This is not required by the fastq format but is often the case with short-read data.

  2. Your barcode is always the last string of letters after the final : on every 4th line starting with the first.

If those assumptions hold true, you can do:

awk -F':' 'NR % 4 == 1 {seq=$NF}
     NR % 4 == 2 { $0=$0 seq}1' R1test.fq > R1test_new.fq

This is sort of the same idea as your code, I just removed some unnecessary steps and fixed some issues. The 1 at the end is awk shorthand for "print this line".

Your code didn't work because you cannot set use -F to set the field separator inside your awk code, the -F is an option to the awk binary, and not a feature of the awk language. To change the field separator within awk scripts you would use the FS variable (e.g. BEGIN{FS=":"}). Next, even if you had managed to change the field separator, that would be irrelevant since the line is split before any code is executed. You can only set the separator in a BEGIN{} block. If you set it anywhere else, you also need to tell awk to reparse the line. And anyway, you wanted : as the field separator, not ;.

Caveat:

This will likely break any downstream processing you want to do since the length of the sequence will not match the length of the phred quality scores. Are you really sure this is a good idea?

4
  • Thank you so much for your fast response terdon and your code worked! I am trying to reformat the fastq files so they can be processed using zUMIs (github.com/sdparekh/zUMIs), which appears to request read 1 to have cellular barcodes and read 2 to have cDNA sequences (or vice versa). You are absolutely right, I am not sure whether the post-processed file as such will be able to be work. :( – julia Jun 14 at 17:55
  • @julia ugh, UMIs are a nightmare. I really doubt this will work though. Each assay manufacturer seems to have their own way of doing UMIs (I've seen them as the last N bases of the reads in the R2 file, or as their own, separate R2 file so the reads are in R1 and R3, and a few other choices). However, adding them manually to the sequence is a new one to me. And cDNA? That seems even stranger. Do come over to Bioinformatics if you get stuck, that's where the people who work in the field mostly hang out. – terdon Jun 14 at 17:58
  • Totally agree! Our own sequencing core sometimes gives separate index reads and sometimes just gives the indexes in the header of R1 and R2! I am sure I will get stuck in the downstream processing. Will definitely come back. Thank you! – julia Jun 14 at 18:02
  • I want to give some updates. Simply adding more characters in line 2 indeed broke the FASTQ format since it requires the quality score (line 4) to have the same number of characters as line 2. In this case, I added some pseudo scores back in line 4 corresponding to the 12-letter barcodes using similar awk command: awk 'NR % 4 == 0 { $0=$0 "IIIIIIIIIIII"}1' R1test_new.fq > R1test_new_new.fq – julia Jun 15 at 18:59
2

One way to handle the fasta file requirements is using the GNU sed stream editor.

Here sed is looking at a line that begins with @ and appends the next line to it. Then the last 12 characters of the @ line are appended to the appended line.

sed -Ee '
  /^@/N
  s/(.{12})\n.*/&\1/
' R1test.fq > R1test_new.fq
2
  • Unfortunately, this isn't safe for this file format because a @ can be the first character on other lines as well. It is rare, but possible and allowed by the format. Of course, you have no way of knowing this since it isn't mentioned in the OP. – terdon Jun 15 at 16:57
  • This is a very good point! Can we use ^@FCID? – julia Jun 15 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.