1

I have a sample file below called reads.fq.gz, which looks like so:

@read1 bla bla sample_id=PD170815
ACTGCGACCCAAACCTTGCACATTGGAC
+
%*&3&&&$$&%$%&3
@read2 bla bla sample_id=PD170815
ACTTTGACCCCAAATCGTGGAC
+
&3&&&%#%^%$%$#%$@#$@#
@read3 bla bla sample_id=PD170815
ATTTGACCCAAATGACCCAGT
+
$%^&()^$#@..;:":
@read4 bla bla sample_id=PD170815
TTGGAACGTGACGATGACCAGT
$%^&*&*^$$$#$#$#$

From this file I want to grep the characters that are between @read and the plus sign +

I have another .txt file where I specify which reads I want to grep from reads.fq.gz. For example, If I want to grep the information from read1 and read4 I would have a list.txt file that looks like so:

read1
read4

If I do:

while read p; do zcat reads.fq.gz | grep -Pzo "\@$p(.)*\n[A-Za-z]*\n\+\n"; done < list.txt

I get what I want:

@read1 bla bla sample_id=PD170815
ACTGCGACCCAAACCTTGCACATTGGAC
+
@read4 bla bla sample_id=PD170815
TTGGAACGTGACGATGACCAGT
+

Though this is a sample file. The file from which I want to grep is huge (23Gb) and the lines between @read and the plus sign + can be very long. So when I run the same command on my original file I get

grep: exceeded PCRE's line length limit

For each entry in list.txt.

Is there a way to specify something like 'no line length limit' for grep so that I can grep these very long lines?

3
  • 1
    I'd avoid multiline matching altogether and use something stateful like awk -vp="$p" '$0 ~ "^@"p {b=1} b{print} /^\+/{b=0}' Nov 5 '19 at 12:41
  • 2
    in fact, if you use awk you can avoid the shell loop as well I think: zcat reads.fq.gz | awk 'NR==FNR {a["@"$0]; next} $1 in a {b=1} b {print} /^\+/ {b=0}' list.txt - Nov 5 '19 at 12:50
  • Excellent, it does the job, thanks very much
    – BCArg
    Nov 5 '19 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.