I'm trying to parse a large file where every 2 consecutive lines have the same length (the text is completely different). I've searched, and my first post here. I found a script and tried to modify it but no joy. File is a sequencing output file. I have already parsed out the sequence, and quality scores, so the file looks like this:
CCTCGNAACCCAAAAACTTTGATTTCTNATAAGGTGCCAGCGGAGTCCTAAAAGCAACATCCGCTGATCCCTGGT
AAAAA#EEEEEEEEEEEEEEEEEEEEE#EEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
CCCCANCCAAACTCCCCACCTGACAATNTCCTCCGCCCGGATCGACCCGCCGAAGCGAGTCTTGGGTCTAAA
AAAAA#EEEEEEEEEEEAEEEEEEEEE#EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
ATCGTNTATGGTTGAGACTAGGACGGTNTCTGATCGTCTTCGAGCCCCCAACTTTCGTTCTTGATTAATGAAAAC
AAAAA#EEEEEEEEEEEEEEEEEEEEE#EEEEEEEEEEEEAEEEEEEAEEEAEEEEEEEEEEEEEEEEEEEEEEE
CCCACNTGGAGCTCTCGATTCCGTGGGNTGGCTCAACAAAGCAGCCACCCCGTCCTACCTATTTAAAGTTTG
AAAAA#EEEEEEEEEEEEEEEEEEEEE#EEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEEEEEEEEEEEEEE
GCATCNTTTATGGTTGAGACTAGGACGNTATCTGATCGTCTTCGAGCCCCCAACTTTCGTTCTTGATTAATGAA
6AAAA#EEEEEAAAEEEEEEAEEAEEE#EEEEEEEAEAEEEEAEEAAA/EAEEEEAEEAEEAEEAEAAEEEEEE
The problem: Somewhere there is a corrupted pair of lines such that each sequence base, does not have a corresponding score, i.e. the lengths of every pair of two lines should be equal, how can I parse out the pair that is incorrect? File is 100-million lines.
I tried this code named parser.sh:
{ curr = $0 }
(NR%2)==0 {
currLgth = length(curr)
prevLgth = length(prev)
maxLgth = (currLgth > prevLgth ? currLgth : prevLgth)
if (prevLgth==currLgth) {
print ""
print prevLgth
print currLgth
for (i=1; i<=maxLgth; i++) {
}
}
}
{ prev = curr }
and would run awk -f parser.sh filename
but this printed out all the lines lengths even though I was using "not equal" ('==').
75
75
72
72
75
75
72
72
Am not a coder, so apologies in advance, but need help with this. Usually can find code and modify it to work, but not in this instance. -p
Fastq files have four lines for each read. Read#1 e,g, will have the following 4 lines:
@sample1
CGGCATCGTTTATGGTTGAGACTAGGACG
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEE
The first line is the sample name, the second line is the actual sequence, the third line is a '+' symbol and the fourth line is a set of ASCII "scores" for each base in the sequence. Each base has exactly one score, hence the length of line 2 must equal the length of line four. I had parsed out lines 2 and 4, looking for pairs of line with unequal length. Instead I got what looks like the pairing was lost.
Here's an example of what a FASTQ file might look like, with the question marks representing the lost or unparsed quality scores:
@sample1
CGGCATCGTTTATGGTTGAGACTAGGACG
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEE
@sample2
CCGGCTTCCGGTTCATCCCGCATCGCCAGTTC
+
@sample3
AAAA6E6/EEEEEEEE6/EE/EEAEEAA//E/
+
@sample4
ATTTCGGGGGGGGGGGGGG
+
??????????????????????????????????
@Sample5
GGTTAGCGCGCAGTTGGGCACCGTAACCCGGCTT
+
AAAAAEEEEEAEEEEEEEEEEEEEEEEEE//<EE
@sample6
CTAACCTGTCTCACGACGGTCTAAACCCAGCTCA
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
Here is what my (line2 + line4) parsed files looked like:
CGGCATCGTTTATGGTTGAGACTAGGACG
AAAAAEEEEEEEEEEEEEEEEEEEEEEEE
CCGGCTTCCGGTTCATCCCGCATCGCCAGTTC
AAAA6E6/EEEEEEEE6/EE/EEAEEAA//E/
ATTTCGGGGGGGGGGGGGG
GGTTAGCGCGCAGTTGGGCACCGTAACCCGGCTT
AAAAAEEEEEAEEEEEEEEEEEEEEEEEE//<EE
CTAACCTGTCTCACGACGGTCTAAACCCAGCTCA
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
There are consecutive two sequence lines with no quality score line between them:
ATTTCGGGGGGGGGGGGGG
GGTTAGCGCGCAGTTGGGCACCGTAACCCGGCTT
Using the code you gave me:
awk 'NR%2==0 && length($0)!=last{print "Bad pair at lines",NR-1,"and",NR}{last=length($0)}' Fastq-seq-qual-parsed.txt
Bad pair at lines 5 and 6
OR: ./new-try.awk
.sh
extension on a file containing awk code is confusing.@sample2
and@sample3
are missing scores completely. Is this what you're working with?