Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to capture a string and the 21 characters preceding it then output this to a new file. This is what I'm using currently:

    grep -o ".\{21\}gt" ../data/fastadata.txt > primerdata.txt

And I get some of the output I desire, but it skips instances of gt, when any amount of the preceding 21 characters were part of a previous capture string. For example:

aaaatccataaatcgaggattacaagtggaaaacaaggaggcagt

will capture

tccataaatcgaggattacaagt

but not

caagtggaaaacaaggaggcagt

I believe it's, because the two strings would share caagt and the end and beginning of strings 1 & 2 respectively. It ends up missing at least half of all instances.

How do I correct this?

It captures 121 instances of ...gt, but there are something like 200 instances of 21 characters preceding gt in the text file below.

Here is some text from the file I'm searching that contains instances of what I'm talking about (hard-wraps for formatting):

aaaccggcctcaagggaacgggtatgtctgcctcacctgtcggagatctacccaatcccagtctgcatct
aacggacactctaatgcaactgctggactgctgcttcctcaccctaacctgcagtggccaaatcgttttg
gtatccaccagcgtggagcagctattgggtcactgtcagtccgatttgtatggccagaatctactgcaga
tcacgcatcccgatgatcaggatctgttaagacagcagctaatacccagggatatagagaccctgttcta
tcagcatcagcaccaccagcagcaggggcacaatccccagcagcactccacttccacgtcggcctcaact
tcgggcagtgatctggaggaggaggaaatggagacggaggaacaccgtctgggtcggcagcagggagagg
cggacgatgacgaggatcacccgtacaaccgacgaacacccagcccgcggagaatggcccatttggcgac
cattgatgaccgactacgcatggatcggcgctgctttaccgtccgcttggctagggcttccacgcgagcg
gaggccacgcgtcattacgagcgggttaagatcgatggctgctttcgtcgcagtgactcctccttaaccg
gaggtgccgctgccaactatccgattgtctcccagctgatacgacgctcgagaaacaacaatatgctggc
tgctgctgcagcagtggcagcagaagcggcgacggtgccgccccagcacgatgccattgcccaggcggcg
ctgcacgggattagcggcaatgatattgtcctggtggccatggccagggtgctgcgagaggaacggccgc
ctgaggagacggagggtacagtgggcttgaccatttacagacagccagaaccctatcagttggagtacca
tacgaggcatctaatcgacggcagcatcatcgactgtgatcaaaggattggtctggtggcgggatatatg
aaggatgaggtgggtatattaacatcatctctctgaactgcttacgacaactaatcgtgtactctccact
cgaaacaggtgcgcaaccttagtcccttctgtttcatgcacctggacgacgttcgctgggtgattgtggc
ccttcgacaaatgtacgattgcaacagtgactacggcgagagctgctaccgtctgctgtcccgcaacggg
cgcttcatttacctgcacaccaagggatttctggaggtcgaccgtggcagtaataaggtgcattcctttc
tgtgcgtcaacacgctgctcgatgaggaggcgggccggcaaaaggtgcaggagatgaaggagaaattctc
gacaatcatcaaggcggagatgcccacgcagagcagcagtcccgatttgcccgcctcgcaggcaccgcag
caacttgagagaattgtcctctatctaatagagaacctacagaagagtgtggattcagcagagacggttg
gcggccagggcatggaaagcctaatggacgatggctacagttcgccagcaaataccttaactctcgagga
gttagctccctcgcccacgcccgccttggccttggtgccgccggctccctcatcggtcaagagctccatc
tccaagtcggtgagtgtggtcaatgtgacggcggccagaaagtttcagcaggagcatcagaagcagcgtg
aacgtgaccgtgagcagcttaaggagcgcaccaactccacgcagggcgtgatccggcaactgagcagctg
cctaagcgaggcggaaacggcatcctgtatcctatcaccagccagtagcttgagtgccagcgaagcaccg
gacacgcccgatccgcacagcaacacatcaccgccaccgtcgctccacacacgtcccagtgtcctgcatc
gaaccctgaccagcacgctgcgatgacgggctgatggaacctggtttgccttctaattgggtgtgtggaa
atggacgtcattggtagctcacgtgcccacaaacgaattagtatcggtaatataatcctggccaatcgca
aaatgaaaacccaaaatgtatcagaaaaaaacgagcattattcaaatagtttaaaaattcagccaaaaaa
cttaaaaacgaaaaaaaagagcgtgggttgaaaaaccttttgttttcatattcacatttccaagctttga
gcaatcaaacaattttaattttcagtatacacatatgtataatgagttggctttacaaaagctattaaca
aatcaagcaattgtgt
share|improve this question
1  
Actually, the issue is that it appears to simply stop matching when it reaches the first instance of gt (which should be fairly unsurprising). Do you have any other separations between the string and its surroundings? E.g., could you use anchors (^ and $) to help delineate where the string begins and ends? It would also be helpful for you to paste (in a code block) the input file so folks here can know exactly what you're dealing with. –  HalosGhost Jul 8 at 21:05
    
Could you give a more compact example, that makes a simple test case? –  Volker Siegel Jul 8 at 23:01

1 Answer 1

I think the issue is that the regex consumes the characters that it matches. You may be able to work around that to some extent using zero-length assertions, if your regex engine supports them.

For example, if you only need to count the occurrences, you could use a PCRE consisting of a single character followed by a lookahead consisting of (21 - 1) characters followed by gt, e.g. in GNU grep with its PCRE mode,

$ printf 'aaaatccataaatcgaggattacaagtggaaaacaaggaggcagt' | 
grep -Po '.(?=.{20}gt)'
t
c

which outputs the initial character of each matching substring, allowing overlaps of all but a single character. (You can pipe the result to wc -l for an actual count).

If you need to recover the actual matching strings that is obviously harder, since you need to both consume and not consume characters - probably there is a way to do it by returning the match indices and then doing substring matches, for example something like this in (probably bad) perl

$ printf 'aaaatccataaatcgaggattacaagtggaaaacaaggaggcagt' | 
perl -lne 'while ($_ =~ m/.(?=.{20}gt)/g) {print substr($_,@+[0]-1,23)}'
tccataaatcgaggattacaagt
caagtggaaaacaaggaggcagt
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.