3

I have a program which prints out lines of text ("paragraphs") separated by '--'. For example it might print

--
are you happy
--
I am hungry
are you
--
are(you hungry
too

I want to pipe that into another program (sed maybe?) and get back just the paragraphs that start with a given word (e.g. "are"). So in the above case getting paragraphs that begin with "are" back I'd get

--
are you happy
--
are(you hungry
too

The program prints out a potentially very large number of "paragraphs" but I expect only a small number to match, which is why I would prefer to be able to filter the program's output in a streaming way (avoiding writing everything to a huge file and then filtering it).

3

AWK

Using GNU awk or mawk:

$ awk '$1~"^"word{printf("--\n%s",$0)}' word='are' RS='--\n' infile
--
are you happy
--
are(you hungry
too

This sets the variable word to the word to match at the beginning of the record and RS (record separator) to '--' followed by a new line \n. Then, for any record which starts with the word to match ($1~"^"word) print a formatted record. The format is a starting '--' with a new line with the exact record found.

GREP

Using (GNU for the -z option) grep:

grep -Pz -- '--\nare(?:[^\n]*\n)+?(?=--|\Z)' infile
grep -Pz -- '(?s)--\nare.*?(?=\n--|\Z)\n' infile
grep -Pz -- '(?s)--\nare(?:(?!\n--).)*\n' infile

Description(s) For the following descriptions, the PCRE option (?x) is used to add (a lot) of explaining comments (and spaces) inline with the actual (working) regex. If the comments (and most spaces) (up to the next newline) are removed, the resulting string is still the same regex. This allow the description of the regex in detail in working code. This makes code maintenance a lot easier.

Option 1 regex (?x)--\nare(?:[^\n]*\n)+?(?=--|\Z)

(?x)   # match the remainder of the pattern with the following
       # effective flags: x
       #      x modifier: extended. Spaces and text after a # 
       #      in the pattern are ignored
--     # matches the characters -- literally (case sensitive)
\n     # matches a line-feed (newline) character (ASCII 10)
are    # matches the characters are literally (case sensitive)
(?:    #      Non-Capturing Group (?:[^\n]*\n)+?
[^\n]  #           matches non-newline characters
*      #           Quantifier — Matches between zero and unlimited times, as
       #           many times as possible, giving back as needed (greedy)
\n     #           matches a line-feed (newline) character (ASCII 10)
)      #      Close the Non-Capturing Group
+?     # Quantifier — Matches between one and unlimited times, as
       # few times as possible, expanding as needed (lazy)
       # A repeated capturing group will only capture the last iteration.
       # Put a capturing group around the repeated group to capture all
       # iterations or use a non-capturing group instead if you're not
       # interested in the data
(?=    # Positive Lookahead (?=--|\Z)
       # Assert that the Regex below matches
       #      1st Alternative --
--     #           matches the characters -- literally (case sensitive)
|      #      2nd Alternative \Z
\Z     #           \Z asserts position at the end of the string, or before
       #           the line terminator right at the end of the 
       #           string (if any)
)      #      Closing the lookahead.

Option 2 regex (?sx)--\nare.*?(?=\n--|\Z)\n

(?sx)  # match the remainder of the pattern with the following eff. flags: sx
       #        s modifier: single line. Dot matches newline characters
       #        x modifier: extended. Spaces and text after a # in 
       #        the pattern are ignored
--     # matches the characters -- literally (case sensitive)
\n     # matches a line-feed (newline) character (ASCII 10)
are    # matches the characters are literally (case sensitive)
.*?    # matches any character 
       #        Quantifier — Matches between zero and unlimited times,
       #        as few times as possible, expanding as needed (lazy).
(?=    # Positive Lookahead (?=\n--|\Z)
       # Assert that the Regex below matches
       #        1st Alternative \n--
\n     #               matches a line-feed (newline) character (ASCII 10)
--     #               matches the characters -- literally.
|      #        2nd Alternative \Z
\Z     #               \Z asserts position at the end of the string, or
       #               before the line terminator right at
       #               the end of the string (if any)
)      # Close the lookahead parenthesis.
\n     #        matches a line-feed (newline) character (ASCII 10)

Option 3 regex (?xs)--\nare(?:(?!\n--).)*\n

(?xs)  # match the remainder of the pattern with the following eff. flags: xs
       # modifier x : extended. Spaces and text after a # in are ignored
       # modifier s : single line. Dot matches newline characters
--     # matches the characters -- literally (case sensitive)
\n     # matches a line-feed (newline) character (ASCII 10)
are    # matches the characters are literally (case sensitive)
(?:    # Non-capturing group (?:(?!\n--).)
(?!    #      Negative Lookahead (?!\n--)
       #           Assert that the Regex below does not match
\n     #                matches a line-feed (newline) character (ASCII 10)
--     #                matches the characters -- literally
)      #      Close Negative lookahead
.      #      matches any character
)      # Close the Non-Capturing group.
*      # Quantifier — Matches between zero and unlimited times, as many
       # times as possible, giving back as needed (greedy)
\n     # matches a line-feed (newline) character (ASCII 10)

sed

$ sed -nEe 'bend
            :start  ;N;/^--\nare/!b
            :loop   ;/^--$/!{p;n;bloop}
            :end    ;/^--$/bstart'           infile
1

Using GNU awk or mawk:

$ awk -v word="are" -v RS='--\n' -v ORS='--\n' '$1 ~ "^" word "[[:punct:]]?"' file
are you happy
--
are(you hungry
too
--

This sets the record separator for both input and output to -- followed by a newline. Each paragraph's first word is found in $1. We match this against the given word (potentially followed by a punctuation character). If they match, the paragraph is printed.

Note that the paragraph markers in the output will be put at the end of each paragraph rather than at the start since we use ORS to output them.


Using a sed script:

:top
/^--/!d;                   # This is not a new paragraph, delete
N;                         # Append next line
/^--\nare[[:punct:]]?/!d;  # This is not a paragraph we want, delete
:record
n;                         # Output line, get next
/^--/!brecord;             # Not yet done with this record, branch to :record
btop;                      # Branch to :top

Running:

$ sed -E -f script.sed file
--
are you happy
--
are(you hungry
too

Or, as a one-liner that uses the shell variable $word:

sed -E -e ':t;/^--/!d;N;' \
       -e "/^--\n$word[[:punct:]]?/!d" \
       -e ':r;n;/^--/!br;bt' file
  • Thank you! Is there a way to get this working where the first word can be ended by punctuation? E.g. a paragraph starting "are(you" should be included still? – LangeHaare Jul 20 '18 at 16:15
  • @LangeHaare I saw your modified data and have updated. – Kusalananda Jul 20 '18 at 16:17
  • @Kusalananda I bow to your superior AWK mindset ;-). – Stephen Kitt Jul 20 '18 at 16:26
  • 1
    Your edit makes the output print -- at the end of each record, while the question has the -- at the start of each record. Maybe using a printf("--\n%s",$0) is one possible solution (note that that doesn't use the OFS and that it could be left at the default value). – Isaac Jul 21 '18 at 6:34
  • @Isaac I know. The record separator is by default outputted after the record, to separate it from the next record. The input data uses "human" record separators in that they precede the records. Going with your recommendation, I would more or less duplicate your answer, which I don't want to do. I'm also working on an alternative, which is why I'm not deleting this one. – Kusalananda Jul 21 '18 at 6:57
1

It's an old question, I know, but it hurts me to see all this looping and branching and pattern juggling when a simple

sed '/^--$/!{H;$!d;};x;/^--\nare/!d'

does the same in a natural way.

sed is a linewise stream editor; so if you need a multi-line thing, collect those lines in the hold space with H and on the paragraph mark (^--$) exchange buffers and test whether to print the paragraph (^--\nare means one -- line followed by a line starting with are). The x already preloads the hold space with the paragraph mark.

You don't need GNU tools with wild extensions, you don't need programming skills, just engage with the sed way.

0
perl -l -0777ne 'print $& while /^--\nare\b.*?(?=\n(?:--|\z))/msg' inp.file

perl -F'/^--\n/m' -na0777e 'print "--\n$_" for  grep { /^are\b/ } @F' inp.file

sed -e '
 /--/!{H;$!d;s/.*/--/;}
 1d;x;s/.//
 /^are\>/!{s/.*//;h;d;}
 H;s/.*//;x
' inp.file

I'll come back later for working explanations..

0

Looking at your question I felt like it should be possible to solve it using grep + PCRE's.

  • The #1 method solves that, thanks to @issac for the assistance in hammering it out.
  • The #2 method shows how you can use the inline modifier ((?s)) and lookahead (?!...).
  • My original solution (#3) works well in most situations, except for the types that I highlight below in its section.

grep method #1

$ grep -Pzo -- '--\nare([^\n]*\n)+?(?=--|\Z)' afile

How it works

grep switches
  • -P - PCRE extensions enabled
  • -z - treats input as multi-line, using NUL in place of \n (newlines)
  • -o - only show matches
regex
  • --\nare([^\n]*\n)+?(?=--|\Z)
    • matches the double dashes followed by an are and then a continuation of zero or more non-newline characters -or- a newline.
    • The +? will match 1 or more, but is non-greedy, therefore it won't continue aggressively.
    • Finally the (?=--|\Z) guards for the end of the block looking for either the next double dash -- or end of file (\Z).

grep method #2

This method uses the DOTALL inline modifier to get . to match newlines (`n`).

$ grep -Pzo -- '(?s)--\nare((?!\n--).)+\n' afile

How it works

grep switches
  • -P - PCRE extensions enabled
  • -z - treats input as multi-line, using NUL in place of \n (newlines)
  • -o - only show matches
regex
  • (?s) - inline modifier DOTALL - alls dot to match newline
  • --\nare - matches a newline followed by are
  • ((?!\n--).)+\n - matches characters . as long as the lookahead (?!\n--) doesn't encounter a \n--. This entire block of matching needs to be at least one or more (+) and followed by a newline, \n.

grep method #3 (original)

Here's a grep solution that utilizes the PCRE extension (-P).. This method worked for all the examples provided, it would fail with examples like this:

--
are
some-other-dasher

But otherwise works in most cases I could conceive having to contend with.

$ grep -Pzo -- '--\nare[^\r\n]+[^-]+' afile
--
are you happy

--
are(you hungry
too

How it works

grep switches
  • -P - PCRE extensions enabled
  • -z - treats input as multi-line, using NUL in place of \n (newlines)
  • -o - only show matches
regex
  • '--\nare[^\r\n]+[^-]+'
    • matches the double dashes followed by a newline and the word are.
    • It will then continue printing the rest of the line that are is on until a newline is encountered.
    • It then prints characters until a sequence of dashes is encountered.

References

  • Use a record with a first line of are(you hungry and a second line of too-maybe? – Isaac Jul 21 '18 at 5:52
  • @Isaac - so close, I tried the dashes on the are lines but hadn't put one into the lines that came after it. The ? doesn't really come into play, you're just throwing extra chars to make it more complicated? – slm Jul 21 '18 at 6:17
  • No @slm, in no way trying to confuse or anything, it is just that the sentence is a question, it starts with an Are :-) ..... – Isaac Jul 21 '18 at 6:20
  • @Isaac - gotcha. Can't come up w/ a regex to thwart your dash in secondary lines. I felt like a lookahead should be able to deal w/ it but I couldn't find one that worked here. Couldn't find anything that broke the awk solution you presented either 8-(. – slm Jul 21 '18 at 6:22
  • Maybe: grep -Pzo -- '--\nare([^\n]*\n)+?(?=--|\Z)' infile :-P – Isaac Jul 21 '18 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.