Grepping string, but include all non-blank lines following each grep match

Question

Consider the following toy example:

this is a line 
this line contains FOO 
this line is not blank

This line also contains FOO

Some random text

This line contains FOO too
Not blank 
Also not blank

More random text 
FOO!
Yet more random text
FOO!

So, I want the results of a grep for FOO, but with the extra wrinkle that lines following the matching lines should be included, as long as they are not blank, and they do not themselves contain FOO. So the matches would look as follows, with the different matches separated:

MATCH 1

this line contains FOO 
this line is not blank

MATCH 2

This line also contains FOO

MATCH 3

This line contains FOO too 
Not blank 
Also not blank

MATCH 4

FOO!
Yet more random text

MATCH 5

FOO!

Bonus points (metaphorically speaking) for a simple single line script that can be run on the command line.

ADDENDUM: Adding a running count of the match number would be quite handy, if it is not too hard.

Answer 1

Using awk rather than grep:

awk '/FOO/ { if (matching) printf("\n"); matching = 1 }
     /^$/  { if (matching) printf("\n"); matching = 0 }
     matching' file

A version that enumerates the matches:

awk 'function flush_print_maybe() {
         if (matching) printf("Match %d\n%s\n\n", ++n, buf)
         buf = ""
     }
     /FOO/ { flush_print_maybe(); matching = 1 }
     /^$/  { flush_print_maybe(); matching = 0 }
     matching { buf = (buf == "" ? $0 : buf ORS $0) }
     END   { flush_print_maybe() }' file

Both awk programs uses a very simple "state machine" to determine if it's currently matching or not matching. A match of the pattern FOO will cause it to enter the matching state, and a match of the pattern ^$ (an empty line) will cause it to enter the non-matching state.

Output of empty lines between matching sets of data happens at state transitions from matching (either into matching or into non-matching).

The first program prints any line when in the matching state.

The second program collects lines in a buf variable when in a matching state. It flushes (empties) this after possibly printing it (depending on the state), together with a Match N label at state transitions (when the first program would output an empty line).

Output of this last program on the sample data:

Match 1
this line contains FOO
this line is not blank

Match 2
This line also contains FOO

Match 3
This line contains FOO too
Not blank
Also not blank

Match 4
FOO!
Yet more random text

Match 5
FOO!

Answer 2

sed -ne '/FOO/{x;P;x};/FOO/,/^$/p' testfile

Each block of non-empty lines in the output is a single chunk of matched data from the input. The number of newlines varies.

This

1. suppresses output (-n); then
2. prints a blank line before every occurrence of "FOO" (/FOO/{x;P;x} - uses the empty hold space);
3. selects ranges of lines starting from ones containing FOO (/FOO/) and ending on empty lines (/^$/); and finally
4. prints those lines (p).

---

this line contains FOO
this line is not blank


This line also contains FOO


This line contains FOO too
Not blank
Also not blank


FOO!
Yet more random text

FOO!

Answer 3

I don’t think this is doable with grep, but it is with AWK:

#! /usr/bin/awk -f

/FOO/ {
  matched = 1
  if (notfirst) print ""
  notfirst = 1
}

/^$/ {
  matched = 0
}

matched

With a count of matches:

#! /usr/bin/awk -f

/FOO/ {
  matched = 1
  if (matches) print ""
  printf "Match %d\n", ++matches
}

/^$/ {
  matched = 0
}

matched

In both cases, the first two blocks determine whether the current record should be copied to the output. When the current record matches “FOO”, the first block sets matched to 1, outputs a blank record if necessary (to separate the forthcoming output from the previous match); in the second variant, it also increments the matches counter and outputs a header. When the current record is empty, the second block sets matched to 0. The lonely matched condition prints the current record if matched is 1.

Answer 4

I've included a pcregrep solution and a python solution.

Multiline Grep Solution

If you have pcregrep installed, you could use a multiline pattern, such as ^.*FOO.*$\n?(^.*\S.*$\n?)*, e.g.:

pcregrep -M '^.*FOO.*$\n?(^.*\S.*$\n?)*' test.txt

The subexpression ^.*FOO.*$\n? will match any line containing the string FOO and the subexpression (^.*\S.*$\n?)* will match any number of subsequent lines that containing a non-whitespace character.

Python Solution

Here is a Python script that should do what you want:

#!/usr/bin/env python3
# -*- encoding: utf8 -*-
"""grep_follow.py

Search a text file for a pattern,
and output that pattern and the
non-empty lines which immediately follow it.
"""

import re
import sys

# Get the search pattern and the input file as command-line arguments
pattern = sys.argv[1]
input_file = sys.argv[2]

# Set a flag to determine whether or not to output the current line
print_flag = False

with open(input_file, "r") as _input_file:

    # Iterate over the lines of the input file
    for line in _input_file:

        # Remove trailing whitespace
        line = line.rstrip()

        # If the line is empty, stop producing output
        if not line.strip():
            print_flag = False

        # If the line matches the search pattern, start producing output
        elif re.search(pattern, line):
            print_flag = True

        # If the print flag is set then output the line
        if print_flag:
            print(line)

You would run it like this:

$ python grep_follow.py FOO test.txt

this line contains FOO
this line is not blank
This line also contains FOO
This line contains FOO too
Not blank
Also not blank
FOO!
Yet more random text
FOO!

Answer 5

awk '/FOO/{print "===match " ++i "==="} /FOO/,/^$/' file

===match 1===
this line contains FOO
this line is not blank

===match 2===
This line also contains FOO

===match 3===
This line contains FOO too
Not blank
Also not blank

===match 4===
FOO!
Yet more random text
===match 5===
FOO!

A similar variant where FOO could be easily changed to something else:

awk -vpat=FOO '$0~pat{print "===match " ++i "==="} $0~pat,/^$/' file

Omitting the terminating empty line from the default print is left as an exercise to the reader ;-)

Answer 6

One way can be using Perl's range operator ...:

$ perl -lne '
     print s|^(?=.*FOO)|$/."MATCH#".++$c.$/|re if /FOO/ ... /^$/;
 ' input.txt

MATCH#1
this line contains FOO 
this line is not blank


MATCH#2
This line also contains FOO


MATCH#3
This line contains FOO too
Not blank 
Also not blank


MATCH#4
FOO!
Yet more random text

MATCH#5
FOO!

In case the extra blank lines are an issue, you can use the below code, where make use of the range operator in scalar context in order to have a peek into the state machine values and perform actions based on them.

Note :

$/ = RS = "\n"

$, = OFS = ""

$ perl -lne '
    print
      s{^(?=.*FOO)}
       {
         ($,,$/)[$r>1] . 
         "MATCH#" . ++$count . $/
       }rex 
          if $r = /FOO/ ... /^$/;
' input.txt

Here's another method to tackle the pbm, this time using the paragraph mode of Perl. We read in records a para at a time and split them on newlines into fields.

Then we make sure to handle only those paras that happen to have the string foo in them. Finally, we print the fields with the constraint on foo-containing fields being treated specially. Even here we deal with the first field differently than the rest.

$ perl -aln -F'\n' -00 -e '
     next unless /FOO/;
     shift @F until $F[0] =~ /FOO/;
     push @F, $, if ! eof;
     print "MATCH#", ++$k, $\, shift @F;
     print for map { /FOO/ and $_ = "\nMATCH#" . ++$k . "\n$_";$_ } @F;
' input.txt 

Answer 7

Here is a one-liner that may not exactly match the criteria, but comes close enough for most purposes.

gawk '/FOO/{p=1} {if(p) print} /^$/{p=0}' infile.txt

First rule sets p=1 if it sees FOO. Second rule prints the current line if p is not zero. Third rule sets p=0 if the line (already printed) is blank. Swap second and third rules if you do not want the blank line terminator to be output.

This is a variant which numbers and separates the matches if there are multiple FOO instances not separated by blank lines

gawk '/FOO/{p=1; n++; print "\n" n} /^$/{p=0} //{if (p) print $0 }' infile.txt