How to get all lines between first and last occurrences of patterns?

Question

How can I trim a file (well input stream) so that I only get the lines ranging from the first occurrence of pattern foo to the last occurrence of pattern bar?

For instance consider the following input :

A line
like
foo
this 
foo
bar
something
something else
foo
bar
and
the
rest

I expect this output:

foo
this 
foo
bar
something
something else
foo
bar

Answer 1

sed -n '/foo/{:a;N;/^\n/s/^\n//;/bar/{p;s/.*//;};ba};'

The sed pattern matching /first/,/second/ reads lines one by one. When some line matches to /first/ it remembers it and looks forward for the first match for the /second/ pattern. In the same time it applies all activities specified for that pattern. After that process starts again and again up to the end of file.

That's not that we need. We need to look up to the last matching of /second/ pattern. Therefore we build construction that looks just for the first entry /foo/. When found the cycle a starts. We add new line to the match buffer with N and check if it matches to the pattern /bar/. If it does, we just print it and clear the match buffer and janyway jump to the begin of cycle with ba.

Also we need to delete newline symbol after buffer clean up with /^\n/s/^\n//. I'm sure there is much better solution, unfortunately it didn't come to my mind.

Hope everything is clear.

Answer 2

I would do it with a little Perl one-liner.

cat <<EOF | perl -ne 'BEGIN { $/ = undef; } print $1 if(/(foo.*bar)/s)'
A line
like
foo
this 
foo
bar
something
something else
foo
bar
and
the
rest
EOF

yields

foo
this 
foo
bar
something
something else
foo
bar

Answer 3

Here's a two-pass GNU sed solution that doesn't require much memory:

< infile                                     \
| sed -n '/foo/ { =; :a; z; N; /bar/=; ba }' \
| sed -n '1p; $p'                            \
| tr '\n' ' '                                \
| sed 's/ /,/; s/ /p/'                       \
| sed -n -f - infile

Explanation

• First sed invocation passes infile and finds first occurrence of foo and all subsequent occurrences of bar.
• These addresses are then shaped into a new sed script with two invocations of sed and one tr. Output of the third sed is [start_address],[end_address]p, without the brackets.
• Final invocation of sed passes the infile again, printing the found addresses and everything in between.

Answer 4

If the input file fits comfortably in memory, keep it simple.

If the input file is huge, you can use csplit to break it into pieces at the first foo and at every subsequent bar then assemble the pieces. The pieces are called piece-000000000, piece-000000001, etc. Choose a prefix (here, piece-) that won't clash with other existing files.

csplit -f piece- -n 9 - '%foo%' '/bar/' '{*}' <input-file

(On non-Linux systems, you'll have to use a large number inside the braces, e.g. {999999999}, and pass the -k option. That number is the number of bar pieces.)

You can assemble all the pieces with cat piece-*, but this will give you everything after the first foo. So remove that last piece first. Since the file names produced by csplit don't contain any special characters, you can work them over without taking any special quoting precaution, e.g. with

rm $(echo piece-* | sed 's/.* //')

or equivalently

rm $(ls piece-* | tail -n 1)

Now you can join all the pieces and remove the temporary files:

cat piece-* >output
rm piece-*

If you want to remove the pieces as they are concatenated to save disk space, do it in a loop:

mv piece-000000000 output
for x in piece-?????????; do
  cat "$x" >>output; rm "$x"
done