as far as i know all unix text processing utilities are reading one line at a time and performing one action on this line.

I have a huge file with a single line of text which contains several tokens im concerned with.

You can think of the content of the file as something like this: xzxzxzzxzxAxzzBxzxCzxxzxxzxzzxzxzAzBzxxxxzzCzxzxzxzxzxxzz

I want to get the two strings between (A and B) and (B and C) for every occurrence of A.*B.*C. In this example my desired output would be this:

xzz xzx

z zxxxxzz

How do i do this?

edit: sorry, i didn't make it clear. A, B and C are long Strings that can only be identified by regular expressions.


I'm sure there are many interesting answers using awk,perl,sed, and others. Here is a rather simplistic options that uses tr to turn this problem back into one a problem that we know how to solve--finding a pattern within a line:

 $ tr 'C' '\n' <test.file | sed -n 's/.*A\(.*\)B\(.*$\)/\1 \2/p'

The tr 'C' '\n' command translates any "C" in the input into a newline character. Thus, it is then necessary to just pipe it into a command that will output the text between A and B and between B and the end of the line.

If A, B, and C are regular expressions rather than simple characters, try:

sed -e 's/C/\n/g' < test.file | sed -n 's/.*A\(.*\)B\(.*\)/\1 \2/p'

This uses the same basic idea, but uses sed to create the newlines.

  • sorry, i didn't mention that A, B and C have to be regular expressions. and afaik tr only works with single characters.
    – Justus1
    Dec 12 '10 at 17:44
  • I updated it with a command that should work even for complex A,B, and C.
    – Steven D
    Dec 12 '10 at 17:54

Awk generalizes the notion of lines to record, which can be terminated by any character. Several implementations, such as Gawk, support an arbitrary regular expression as the record separator. Untested:

gawk -vRS='C' 'sub(/.*A/, "") && sub(/B.*/) {print}'

If the line can fit into memory, then repeated use of the split function from Perl would work. Otherwise, I would read the file in blocks (with the Perl sysread function) and process each block individually as above -- allowing for stings of interest to cross block boundaries.

  • it easily fits into memory. i don't have any experience with perl.
    – Justus1
    Dec 12 '10 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.