Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

How can I trim a file (well input stream) so that I only get the lines ranging from the first occurrence of pattern foo to the last occurrence of pattern bar?

For instance consider the following input :

A line
like
foo
this 
foo
bar
something
something else
foo
bar
and
the
rest

I expect this output:

foo
this 
foo
bar
something
something else
foo
bar
share|improve this question
2  
Single pass stream or a file? This is much easier to do when random access is allowed. With a file you would just find the first foo and last bar and print everything in between, if anything. With a stream you would have to read until the first foo, and buffer all subsequent lines in memory until EOF, flushing the buffer every time a bar is seen. This could mean buffering the entire stream in memory. –  jw013 Sep 12 '12 at 14:25

4 Answers 4

up vote 4 down vote accepted
sed -n '/foo/{:a;N;/^\n/s/^\n//;/bar/{p;s/.*//;};ba};'

The sed pattern matching /first/,/second/ reads lines one by one. When some line matches to /first/ it remembers it and looks forward for the first match for the /second/ pattern. In the same time it applies all activities specified for that pattern. After that process starts again and again up to the end of file.

That's not that we need. We need to look up to the last matching of /second/ pattern. Therefore we build construction that looks just for the first entry /foo/. When found the cycle a starts. We add new line to the match buffer with N and check if it matches to the pattern /bar/. If it does, we just print it and clear the match buffer and janyway jump to the begin of cycle with ba.

Also we need to delete newline symbol after buffer clean up with /^\n/s/^\n//. I'm sure there is much better solution, unfortunately it didn't come to my mind.

Hope everything is clear.

share|improve this answer
    
It works! It would be super cool if you could walk us through the construction of such a command. I'd feel dumb simply copy/pasting it from some website online ;) –  rahmu Sep 12 '12 at 15:41
    
Sorry I didn't post the explanation with the answer. Now it is in the post. –  rush Sep 12 '12 at 15:58

I would do it with a little Perl one-liner.

cat <<EOF | perl -ne 'BEGIN { $/ = undef; } print $1 if(/(foo.*bar)/s)'
A line
like
foo
this 
foo
bar
something
something else
foo
bar
and
the
rest
EOF

yields

foo
this 
foo
bar
something
something else
foo
bar
share|improve this answer
1  
If this were code-golf, you could use E instead of e and -00777 instead of the $/ bit (see perlrun(1)). Which would shorten it to: perl -0777 -nE 'say /(foo.*bar)/s', still sort of readable. –  Thor Sep 12 '12 at 15:51
    
I didn't know about these flags! I am sure that especially -0[octal] will find it's way in my workflow! Thanks for that –  user1146332 Sep 12 '12 at 15:57

Here's a two-pass GNU sed solution that doesn't require much memory:

< infile                                     \
| sed -n '/foo/ { =; :a; z; N; /bar/=; ba }' \
| sed -n '1p; $p'                            \
| tr '\n' ' '                                \
| sed 's/ /,/; s/ /p/'                       \
| sed -n -f - infile

Explanation

  • First sed invocation passes infile and finds first occurrence of foo and all subsequent occurrences of bar.
  • These addresses are then shaped into a new sed script with two invocations of sed and one tr. Output of the third sed is [start_address],[end_address]p, without the brackets.
  • Final invocation of sed passes the infile again, printing the found addresses and everything in between.
share|improve this answer

If the input file fits comfortably in memory, keep it simple.

If the input file is huge, you can use csplit to break it into pieces at the first foo and at every subsequent bar then assemble the pieces. The pieces are called piece-000000000, piece-000000001, etc. Choose a prefix (here, piece-) that won't clash with other existing files.

csplit -f piece- -n 9 - '%foo%' '/bar/' '{*}' <input-file

(On non-Linux systems, you'll have to use a large number inside the braces, e.g. {999999999}, and pass the -k option. That number is the number of bar pieces.)

You can assemble all the pieces with cat piece-*, but this will give you everything after the first foo. So remove that last piece first. Since the file names produced by csplit don't contain any special characters, you can work them over without taking any special quoting precaution, e.g. with

rm $(echo piece-* | sed 's/.* //')

or equivalently

rm $(ls piece-* | tail -n 1)

Now you can join all the pieces and remove the temporary files:

cat piece-* >output
rm piece-*

If you want to remove the pieces as they are concatenated to save disk space, do it in a loop:

mv piece-000000000 output
for x in piece-?????????; do
  cat "$x" >>output; rm "$x"
done
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.