32

It seems I am misusing grep/egrep.

I was trying to search for strings in multiple line and could not find a match while I know that what I'm looking for should match. Originally I thought that my regexes were wrong but I eventually read that these tools operate per line (also my regexes were so trivial it could not be the issue).

So which tool would one use to search patterns across multiple lines?

31

Here's a sed one that will give you grep-like behavior across multiple lines:

sed -n '/foo/{:start /bar/!{N;b start};/your_regex/p}' your_file

How it works

  • -n suppresses the default behavior of printing every line
  • /foo/{} instructs it to match foo and do what comes inside the squigglies to the matching lines. Replace foo with the starting part of the pattern.
  • :start is a branching label to help us keep looping until we find the end to our regex.
  • /bar/!{} will execute what's in the squigglies to the lines that don't match bar. Replace bar with the ending part of the pattern.
  • N appends the next line to the active buffer (sed calls this the pattern space)
  • b start will unconditionally branch to the start label we created earlier so as to keep appending the next line as long as the pattern space doesn't contain bar.
  • /your_regex/p prints the pattern space if it matches your_regex. You should replace your_regex by the whole expression you want to match across multiple lines.
| improve this answer | |
  • 1
    +1 Adding this to the toolikt! Thanks. – wmorrison365 Apr 24 '18 at 15:36
  • Note: On MacOS this gives sed: 1: "/foo/{:start /bar/!{N;b ...": unexpected EOF (pending }'s) – Stan James Sep 20 '18 at 16:32
  • 1
    Getting sed: unterminated { error – Nomaed Apr 29 '19 at 14:07
  • @Nomaed Shot in the dark here, but does your regex happen to contain any "{" characters? If so, you'll need to backslash-escape them. – Joseph R. Apr 29 '19 at 16:37
  • 1
    @Nomaed It seems that it has to do with the differences between sed implementations. I tried to follow the recommendations in that answer to make the above script standard-compliant but it told me that "start" was an undefined label. So I'm not sure whether this can be done in a standard-compliant way. If you do manage it, please feel free to edit my answer. – Joseph R. May 1 '19 at 17:24
21

I generally use a tool called pcregrep which can be installed in most of the linux flavour using yum or apt.

For eg.

Suppose if you have a file named testfile with content

abc blah
blah blah
def blah
blah blah

You can run the following command:

$ pcregrep -M  'abc.*(\n|.)*def' testfile

to do pattern matching across multiple lines.

Moreover, you can do the same with sed as well.

$ sed -e '/abc/,/def/!d' testfile
| improve this answer | |
6

Here's a simpler approach using Perl:

perl -e '$f=join("",<>); print $& if $f=~/foo\nbar.*\n/m' file

or (since JosephR took the sed route, I'll shamelessly steal his suggestion)

perl -n000e 'print $& while /^foo.*\nbar.*\n/mg' file

Explanation

$f=join("",<>); : this reads the entire file and saves it's contents (newlines and all) into the variable $f. We then attempt to match foo\nbar.*\n, and print it if it matches (the special variable $& holds the last match found). The ///m is needed to make the regular expression match across newlines.

The -0 sets the input record separator. Setting this to 00 activates 'paragraph mode' where Perl will use consecutive newlines (\n\n) as the record separator. In cases where there are no consecutive newlines, the entire file is read (slurped) at once.

Warning:

Do not do this for large files, it will load the entire file into memory and that may be a problem.

| improve this answer | |
4

Simply a normal grep which supports Perl-regexp parameter P will do this job.

$ echo 'abc blah
blah blah
def blah
blah blah' | grep -oPz  '(?s)abc.*?def'
abc blah
blah blah
def

(?s) called DOTALL modifier which makes dot in your regex to match not only the characters but also the line breaks.

| improve this answer | |
  • When I try this solution the output does not end at 'def' but goes to the end of the file 'blah' – buckley Oct 25 '19 at 14:43
  • maybe your grep does not support -P option – Avinash Raj Oct 26 '19 at 1:31
3

One way to do this is with Perl. e.g. here's the contents of a file named foo:

foo line 1
bar line 2
foo
foo
foo line 5
foo
bar line 6

Now, here's some Perl which will match against any line that begins with foo followed by any line that begins with bar:

cat foo | perl -e 'while(<>){$all .= $_}
  while($all =~ /^(foo[^\n]*\nbar[^\n]*\n)/m) {
  print $1; $all =~ s/^(foo[^\n]*\nbar[^\n]*\n)//m;
}'

The Perl, broken down:

  • while(<>){$all .= $_} This loads the entire standard input in to the variable $all
  • while($all =~ While the variable all has the regular expression...
  • /^(foo[^\n]*\nbar[^\n]*\n)/m The regex: foo at the beginning of the line, followed by any number of non-newline chars, followed by a newline, followed immediately by "bar", and the rest of the line with bar in it. /m at the end of the regex means "match across multiple lines"
  • print $1 Print the part of the regex that was in parenthesis (in this case, the entire regular expression)
  • s/^(foo[^\n]*\nbar[^\n]*\n)//m Erase the first match for the regex, so we can match multiple cases of the regex in the file in question

And the output:

foo line 1
bar line 2
foo
bar line 6
| improve this answer | |
  • 3
    Just dropped by to say your Perl can be shortened to the more idiomatic: perl -n0777E 'say $& while /^foo.*\nbar.*\n/mg' foo – Joseph R. Feb 2 '14 at 13:10
2

The grep alternative sift supports multiline matching (disclaimer: I am the author).

Suppose testfile contains:

<book>
  <title>Lorem Ipsum</title>
  <description>Lorem ipsum dolor sit amet, consectetur
  adipiscing elit, sed do eiusmod tempor incididunt ut
  labore  et dolore magna aliqua</description>
</book>


sift -m '<description>.*?</description>' (show the lines containing the description)

Result:

testfile:  <description>Lorem ipsum dolor sit amet, consectetur
testfile:  adipiscing elit, sed do eiusmod tempor incididunt ut
testfile:  labore  et dolore magna aliqua</description>


sift -m '<description>(.*?)</description>' --replace 'description="$1"' --no-filename (extract and reformat the description)

Result:

description="Lorem ipsum dolor sit amet, consectetur
  adipiscing elit, sed do eiusmod tempor incididunt ut
  labore  et dolore magna aliqua"
| improve this answer | |
  • 1
    Very nice tool. Congratulations! Try to include it in distributions like Ubuntu. – Lourenco Jan 10 '17 at 20:37
2

I solved this one for me using grep and -A option with another grep.

grep first_line_word -A 1 testfile | grep second_line_word

The -A 1 option prints 1 line after the found line. Of course it depends on your file and word combination. But for me it was the fastest and reliable solution.

| improve this answer | |
  • alias grepp='grep --color=auto -B10 -A20 -i ' then cat somefile | grepp blah | grepp foo | grepp bar ... yes those -A and -B are very handy ... you have the best answer – Scott Stensland Apr 10 '19 at 14:12
  • This isn't super deterministic and it ignores the entire pattern in favor of just getting a different single line (just based on its proximity to the first line). It's better to tell the program to go however far it needs to go in order to get to some sort of pattern you're absolutely sure is the end of the text you're trying to match. For example if testfile gets updated such that second_line_word is on the third line, then not only are you now missing the first line (due to your second grep) but you're not missing the line that started showing up inbetween the two. – Bratchley Apr 9 at 13:23
  • This would be a good enough MO for ad hoc commands where you really do just want a single line in output you already understood though. I don't think that's what the OP is after though and you could probably also just copy/paste at that point due to it being ad hoc. – Bratchley Apr 9 at 13:25
1

Supppose we have the file test.txt containing :

blabla
blabla
foo
here
is the
text
to keep between the 2 patterns
bar
blabla
blabla

The following code can be used :

sed -n '/foo/,/bar/p' test.txt

For the following output :

foo
here
is the
text
to keep between the 2 patterns
bar
| improve this answer | |
1

If we want to get the text between the 2 patterns excluding themselves.

Supppose we have the file test.txt containing :

blabla
blabla
foo
here
is the
text
to keep between the 2 patterns
bar
blabla
blabla

The following code can be used :

 sed -n '/foo/{
 n
 b gotoloop
 :loop
 N
 :gotoloop
 /bar/!{
 h
 b loop
 }
 /bar/{
 g
 p
 }
 }' test.txt

For the following output :

here
is the
text
to keep between the 2 patterns

How does it work, let's make it step by step

  1. /foo/{ is triggered when line contains "foo"
  2. n replace the pattern space with next line, i.e. the word "here"
  3. b gotoloop branch to the label "gotoloop"
  4. :gotoloop defines the label "gotoloop"
  5. /bar/!{ if the pattern doesn't contain "bar"
  6. h replace the hold space with pattern, so "here" is saved in the hold space
  7. b loop branch to the label "loop"
  8. :loop defines the label "loop"
  9. N appends the pattern to the hold space.
    Now hold space contains :
    "here"
    "is the"
  10. :gotoloop We are now at step 4, and loop until a line contains "bar"
  11. /bar/ loop is finished, "bar" has been found, it's the pattern space
  12. g pattern space is replaced with hold space that contains all the lines between "foo" and "bar" that have saved during the main loop
  13. p copy pattern space to standard output

Done !

| improve this answer | |
  • Well done, +1. I usually avoid using these commands by tr'ing the newlines into SOH and performing normal sed commands then replace the newlines. – A.Danischewski Nov 8 '19 at 2:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.