Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

How to get multiple lines out of a file by a regex?

I often would like to get multiple lines / modify multiple lines by a regex. A example case:

I am trying to read part of an XML/SGML file (they are not necessarily well formed or in a predictable syntax, so a regex would be safer than a proper parser. In addition I would like to be able to do this also whit completely unstructured files where just some key words are known.) in a shell script (running on Solaris and Linux).

Example XML:

<tag1>
 <tag2>bar</tag2>
</tag1>
<tag1>
 <tag2>foo</tag2>
</tag1>

From this I would like to read the <tag1> if it contains foo somewhere within it.

A regex like (<tag1>.*?foo.*?</tag1>) should give the right part but tools like grep and sed only work for me with single lines. How can I get

<tag1>
 <tag2>foo</tag2>
</tag1>

in this example?

share|improve this question
3  
Mandatory link –  evilsoup Oct 26 '13 at 15:26
    
@evilsoup Thats true, but my question is not specifically about XML/SGML files, just about any text files. –  Den Apr 27 at 20:32

6 Answers 6

If you have GNU grep installed you could do multiline search by passing in the -P (perl-regex) flag and activating PCRE_DOTALL with (?s)

grep -oP '(?s)<tag1>(?:(?!tag1).)*?foo(?:(?!tag1).)*?</tag1>' file.txt
<tag1>
<tag2>foo</tag2>
</tag1>

If the above doesn't work on your platform, try passing the -z flag in addition, this forces grep to treat NUL as line separator, causing the entire file to look like a single line.

grep -ozP '(?s)<tag1>(?:(?!tag1).)*?foo(?:(?!tag1).)*?</tag1>' file.txt
share|improve this answer
    
This gives no output on my system when run on the OP's example file. –  terdon Oct 21 '13 at 15:24
    
Works for me. +1. Thanks for the (?s) tip –  Nathan Wallace Oct 21 '13 at 15:30
    
@terdon, what version of GNU grep are you running? –  1_CR Oct 21 '13 at 15:42
    
@1_CR (GNU grep) 2.14 on Debian. I copied the OPs example as is (adding only final newline) and ran your grep on it but got no results. –  terdon Oct 21 '13 at 15:47
1  
@slm, i am on pcre 6.6, GNU grep 2.5.1 on RHEL. Do you mind trying grep -ozP instead of grep -oP on your platforms? –  1_CR Oct 21 '13 at 16:51
#begin command block
#append all lines between two addresses to hold space 
    sed -n -f - <<\SCRIPT file.xml
        \|<tag1>|,\|</tag1>|{ H 
#at last line of search block exchange hold and pattern space 
            \|</tag1>|{ x
#if not conditional ;  clear buffer ; branch to script end
                \|<tag2>[^<]*foo[^\n]*</tag2>|!{s/.*//;h;b}
#do work ; print result; clear buffer ; close blocks
    s?*?*?;p;s/.*//;h;b}}
SCRIPT

If you do the above, given the data you show, before that last cleanup line there, you should be working with a sed pattern space that looks like:

 ^\n<tag1>\n<tag2>foo</tag2>\n</tag1>$

You can print out your pattern space whenever you like with look. You can then address on \n characters.

sed l <file

Will show you each line sed processes it at the stage in which l is called.

So I've just tested it and it needed one more \backslash after the ,comma in the first line, but otherwise works as is. Here I put it in a _sed_function so I can easily call it for demonstration purposes throughout this answer: (works with comments included, but are here removed for brevity's sake)

_sed_function() { sed -n -f /dev/fd/3 
} 3<<\SCRIPT <<\FILE 
    \|<tag1>|,\|</tag1>|{ H
        \|</tag1>|{ x
            \|<tag2>[^<]*foo[^\n]*</tag2>|!{s/.*//;h;b}
    s?*?*?;p;s/.*//;h;b}}
#END
SCRIPT
<tag1>
 <tag2>bar</tag2>
</tag1>
<tag1>
 <tag2>foo</tag2>
</tag1>
FILE


_sed_function
#OUTPUT#
<tag1>
 <tag2>foo</tag2>
</tag1>

Now we'll switch the p for an l so we can see what we're working with as we develop our script and remove the non-op demo s? so the last line of our sed 3<<\SCRIPT just looks like:

l;s/.*//;h;b}}

Then I'll run it again:

_sed_function
#OUTPUT#
\n<tag1>\n <tag2>foo</tag2>\n</tag1>$

Ok! So I was right - that's a good feeling. Now, let's shuffle our look around to see the lines it pulls in but deletes. We'll remove our current l and add one to the !{block} so it looks like:

!{l;s/.*//;h;b}

_sed_function
#OUTPUT#
\n<tag1>\n <tag2>bar</tag2>\n</tag1>$

That's what it looks like just before we wipe it out.

One last thing I want to show you is the Hold space as we build it up. There are a couple of key concepts I hope I can demonstrate. So I remove the last look again and alter the first line to add a peek into Hold space at the end:

{ H ; x ; l ; x

_sed_function
#OUTPUT#
\n<tag1>$
\n<tag1>\n <tag2>bar</tag2>$
\n<tag1>\n <tag2>bar</tag2>\n</tag1>$
\n<tag1>$
\n<tag1>\n <tag2>foo</tag2>$
\n<tag1>\n <tag2>foo</tag2>\n</tag1>$

Hold space survives line cycles - hence the name. So what people often trip up on - ok, what I often trip up on - is that it needs deleting after you use it. In this case I only exchange once, so the hold space becomes the pattern space and vice-versa and this change also survives line cycles.

The effect is that I need to delete my hold space which used to be my pattern space. I do this by first clearing the current pattern space with:

s/.*//

Which simply selects every character and removes it. I cannot use d because this would end my current line cycle and the next command would not complete, which would pretty much trash my script.

h

This works in a similar way to H but it overwrites hold space, so I've just copied my blank pattern space over the top of my hold space, effectively deleting it. Now I can just:

b

out.

And that's how I write sed scripts.

share|improve this answer
    
Welcome to 3k!! –  slm May 1 at 22:59
    
Thanks @slm! Youre a really ok guy, you know that? –  mikeserv May 1 at 23:22
    
Thanks, nice job, very quick ascent to 3k, next up 5k 8-) –  slm May 1 at 23:34
    
I dunno, @slm. Im beginning to see im learning less and less here - maybe ive outgrown its usefulness. I gotta think about it. ive barely even come to the site last couple weeks. –  mikeserv May 1 at 23:36
1  
Well, @slm - youre a rare breed anyway. I do agree about the multiple answers though. Thats why it bugs me when some qs get closed. But that rarely happens, actually. Thanks again, slm. –  mikeserv May 1 at 23:46

@jamespfinn's answer will work perfectly well if your file is as simple as your example. If you have a more complex situation where <tag1> could span more than 2 lines, you'll need a slightly more complex trick. For example:

$ cat foo.xml
<tag1>
 <tag2>bar</tag2>
 <tag3>baz</tag3>
</tag1>
<tag1>

 <tag2>foo</tag2>
</tag1>

<tag1>
 <tag2>bar</tag2>

 <tag2>foo</tag2>
 <tag3>baz</tag3>
</tag1>
$ perl -ne 'if(/<tag1>/){$a=1;} 
            if($a==1){push @l,$_}
            if(/<\/tag1>/){
              if(grep {/foo/} @l){print "@l";}
               $a=0; @l=()
            }' foo.xml
<tag1>

  <tag2>foo</tag2>
 </tag1>
<tag1>
  <tag2>bar</tag2>

  <tag2>foo</tag2>
  <tag3>baz</tag3>
 </tag1>

The perl script will process each line of your input file and

  • if(/<tag1>/){$a=1;} : the variable $a is set to 1 if an opening tag (<tag1>) is found.

  • if($a==1){push @l,$_} : for each line, if $a is 1, add that line to the array @l.

  • if(/<\/tag1>/) : if the current line matches the closing tag:

    • if(grep {/foo/} @l){print "@l"} : if any of the lines saved in the array @l (these are the lines between <tag1> and </tag1>) matches the string foo, print the contents of @l.
    • $a=0; @l=() : empty the list (@l=()) and set $a back to 0.
share|improve this answer
    
This works well except in the case where there are more than one <tag1> containing "foo". In that case it prints every thing from the beginning of the first <tag1> to the end of the last </tag1>... –  Den Oct 23 '13 at 12:55
    
@den I tested it with the example shown in my answer which contains 3 <tag1> with foo and it works fine. When does it fail for you? –  terdon Oct 23 '13 at 13:01
    
it feels so wrong parsing xml using regex :) –  Braiam Apr 27 at 23:58

Here's a sed alternative:

sed -n '/<tag1/{:x N;/<\/tag1/!b x};/foo/p' your_file

Explanation

  • -n means do not print lines unless instructed.
  • /<tag1/ first matches the opening tag
  • :x is a label to enable jumping to this point later
  • N adds the next line to the pattern space (active buffer).
  • /<\/tag1/!b x means if the current pattern space contains no closing tag, branch to the x label created earlier. We thus keep adding lines to the pattern space until we find our closing tag.
  • /foo/p means if the current pattern space matches foo, it should be printed.
share|improve this answer

You could do it with GNU awk I think, by treating the end tag as a record separator e.g. for a known end tag </tag1>:

gawk -vRS="\n</tag1>\n" '/foo/ {printf "%s%s", $0, RT}'

or more generally (with a regex for the end tag)

gawk -vRS="\n</[^>]*>\n" '/foo/ {printf "%s%s", $0, RT}'

Testing it on @terdon's foo.xml:

$ gawk -vRS="\n</[^>]*>\n" '/foo/ {printf "%s%s", $0, RT}' foo.xml
<tag1>

 <tag2>foo</tag2>
</tag1>

<tag1>
 <tag2>bar</tag2>

 <tag2>foo</tag2>
 <tag3>baz</tag3>
</tag1>
share|improve this answer

If your file is structured exactly as you have shown above, you could utilize the -A (lines after) & -B (lines before) flags for grep... for example:

$ cat yourFile.txt 
<tag1>
 <tag2>bar</tag2>
</tag1>
<tag1>
 <tag2>foo</tag2>
</tag1>
$ grep -A1 -B1 bar yourFile.txt 
<tag1>
 <tag2>bar</tag2>
</tag1>
$ grep -A1 -B1 foo yourFile.txt 
<tag1>
 <tag2>foo</tag2>
</tag1>

If your version of grep supports it, you could also use the simpler -C (for context) option which print the surrounding N lines:

$ grep -C 1 bar yourFile.txt 
<tag1>
 <tag2>bar</tag2>
</tag1>
share|improve this answer
    
Thanks, but no. This is only a example and the real stuff looks pretty unpredictable ;-) –  Den Oct 21 '13 at 15:10
1  
That's not finding a tag with foo in it, that's just finding foo and displaying lines of context –  Nathan Wallace Oct 21 '13 at 15:28
    
@NathanWallace yes, which is exactly what the OP was asking for, this answer works perfectly well in the case given in the question. –  terdon Oct 21 '13 at 15:46
    
@terdon that's not at all what the question asks. Quote: "I would like to read the <tag1> if it contains foo somewhere within it." This solution is like "I would like to read 'foo' and 1 line of context regardless of where 'foo' appears". Following your logic, an equally valid answer to this question would be tail -3 input_file.xml. Yes it works for this specific example, but it's not a helpful answer to the question. –  Nathan Wallace Oct 21 '13 at 17:45
    
@NathanWallace my point was that the OP specifically stated this is not a valid XML format, in that case, it could well have been sufficient to print the N lines around the string the OP is searching for. With the information available, this answer was decent enough. –  terdon Oct 21 '13 at 18:25

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.