10

This question already has an answer here:

I want to grab multiple lines in a file with a pattern that has a starting and an ending tag in an ungreedy way.

For example I have the following input:

file.txt

START
test1
test2
foo
END
some
more text

START
test3
bar
test4
test5
END
even more

START
baz
test6
END

Now I want to look for bar and print out everything between START and END, so that I will get:

START
test3
bar
test4
test5
END

What I have so far is the following grep command:

grep -Pzo '(?s)START.*?bar.*?END' file.txt

The problem is, that this expression is greedy and prints out:

START      # starts at first "START"-tag, not the next one
test1      #
test2      #
foo        #
END        #
some       #
more text  #

START
test3
bar
test4
test5
END

It's not done with the grep flags --before-context / --after-context, because the count of lines before and after can differ.

The tool used by text-processing doesn't matter. It should work on a general RedHat system. Also the faster the tool grabs the lines, the better it will be. Because I have big logfiles of each about 150MB.

Can someone tell me, how to achieve my goal the best way?


Update:

Okay, I got it. I just had to think about how to construct my command from don_crisstis link. Here's the solution:

ed -s file.txt <<< $'g/bar/?START?,/END/p\nq\n'

Thank you very much for all your very fast help!

And yes, finally it's a duplicate...

marked as duplicate by don_crissti, Ramesh, taliezin, roaima, Anthon Oct 7 '15 at 17:00

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Do you want to allow for multiple lines within the START and END? It is not entirely clear from your examples as they are all single lines, but you state everything between START and END – DanSut Oct 7 '15 at 15:22
  • 1
    @DanSut The line count before and after can be different. I updated my examples so that it's more clear. – bobbel Oct 7 '15 at 15:29
  • @don_crissti Thanks for showing up this question. Finally, this solved my problem! – bobbel Oct 7 '15 at 15:47
  • Is it really a duplicate if you specifically asked about grep? which you did ;) – DanSut Oct 7 '15 at 16:43
  • 1
    @DanSut I see what you're about to clarify. So, strictly speaking: I'm not aware of the tool used by text-processing for my problem. I also changed the title of my question to satisfy this. Furthermore I added more specific text due to get the fastest solution. – bobbel Oct 7 '15 at 19:42
4

I think your problem is that your non-greedy matches can still swallow up more than you want them to, ie. ENDs and STARTs. This seems to work:

grep -Pzo '(?s)START(?:(?!END).)*?bar(?:(?!START).)*?END' file.txt

It covers all cases in your example and to be complete if you >> file.txt

bar
START
test7
END

Still works.

  • Thanks for your answer! This also works for my specific problem! – bobbel Oct 7 '15 at 16:35
2

I would use awk, where you can specify the record separator. If the record separator is "END" (on its own line), then look for the record containing "bar":

awk 'BEGIN {RS = ORS = "\nEND\n"} /bar/' file.txt

Handling text that appears between the START and END markers. This change feels hacky, but it works for this scenario: using END as the record separator, remove any text before the START keyword

awk '
    BEGIN {RS = ORS = "\nEND\n"} 
    {sub(/^.*\nSTART\n/, "START\n")} 
    /bar/
' file.txt

This may not give desired results if "START" can appear more than once before END

foo
START
hello
START
bar
world
END
baz

will output as

START
bar
world
END
  • Thanks for your answer! But I'm not sure how to match from the START tag on. Between END and START tag there could also be some more unimportant lines that I don't want to match. I've put it in my example what I mean. – bobbel Oct 7 '15 at 15:33
  • Now it works like a charm for my problem! Thank you very much, too! – bobbel Oct 7 '15 at 16:31
1
perl -nE 'BEGIN {$/="\nEND\n"} say /(START.*test.*)/s'

As @bobbel point out, replace say by print to avoid empty line separators.

  • I tried with perl -nE 'BEGIN {$/="\nEND\n"} say /(START.*bar.*)/s' file.txt and got also some new lines after the match. I added a | grep -v ^$ to the end and this solved the problem, too. – bobbel Oct 9 '15 at 12:42
  • Also I did a benchmark of all four solutions (grep, awk, ed, perl) and the perl solution is much faster than all other solutions. Thank you very much! – bobbel Oct 9 '15 at 13:00
  • I replaced say with print to prevent printing out empty new lines. Now it's perfect: perl -nE 'BEGIN {$/="\nEND\n"} print /(START.*bar.*)/s' file.txt – bobbel Oct 9 '15 at 13:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.