Remove lines matching pattern, plus any lines following it matching a different pattern

Question

Let me preface I am not sure if this question has been asked before, I have been Googling for answers but came up short.

I want to use standard Linux/Unix commands (running this on FreeBSD) to exclude lines from a log file that match a pattern. The log file also includes "last message repeated x times" to condense log entries.

As an example, I want to take this:

May 27 2023 11:07 relevant information #1
May 27 2023 11:07 relevant information #2
May 27 2023 11:08 last message repeated 3 times
May 27 2023 11:08 useless information #1
May 27 2023 11:08 last message repeated 5 times
May 27 2023 11:09 last message repeated 8 times
May 27 2023 11:09 relevant information #3
May 27 2023 11:09 useless information #2
May 27 2023 11:10 useless information #3
May 27 2023 11:10 last message repeated 6 times

And get this output:

May 27 2023 11:07 relevant information #1
May 27 2023 11:07 relevant information #2
May 27 2023 11:08 last message repeated 3 times
May 27 2023 11:09 relevant information #3

I've gotten as far as using sed commands to do this, but I don't know enough about how it works to figure it out. I am especially lost when it comes to the log lines that have multiple "last message repeated" following it. Here's what I'm working with currently:

sed '/useless information/{d;N;/last message repeated/d;}' ./logfile.txt

The above first deletes matching lines containing "useless information", then adds the next line to the namespace with N, and then is supposed to delete the resulting line if it contains "last message repeated". But it is only deleting the lines with "useless information".

Answer 1

Using any awk in any shell on every Unix box if you can test for relevant rather than useless information:

awk '/last message repeated/ && f; {f=/relevant information/} f' file
May 27 2023 11:07 relevant information #1
May 27 2023 11:07 relevant information #2
May 27 2023 11:08 last message repeated 3 times
May 27 2023 11:09 relevant information #3

Answer 2

With pcregrep:

$ pcregrep -vM 'useless information(.*\n.*message repeated)*' your-file
May 27 2023 11:07 relevant information #1
May 27 2023 11:07 relevant information #2
May 27 2023 11:08 last message repeated 3 times
May 27 2023 11:09 relevant information #3

-M is pcregrep's Multiline mode which pulls extra lines into the pattern space as needed for the regexp to match (within limits). With that enabled the m perl/PCRE flag (which makes ^ and $ match at the beginning/end of each line within the subject as opposed to just the beginning/end of the subject) is enabled and the s flag (which makes . also match newline characters) is not enabled which explains why those .*s above don't gobble up the whole input despite being greedy.

With sed, that could be for instance:

sed -n '
  :start
  /useless information/ {
    :more
    n
    /message repeated/ b more
    b start
  }
  p'

With awk:

awk '! (useless && /message repeated/ || \
        (useless = /useless information/))'

Note that (...) are needed around the assignment to useless in busybox awk at least.

Same in perl:

perl -ne 'print unless $useless and /message repeated/ or
            $useless = /useless information/'

(using and/or instead of &&/|| as they have lower precedence than = removing the need for parenthesis; see perldoc perlop for reference).

Answer 3

Using Raku (formerly known as Perl_6)

~$ raku -ne '.put unless my $useless and /message \s repeated/ or $useless = /useless \s information/;'  file

#OR

~$ raku -ne '.put unless my $useless and / "message repeated" / or $useless = / "useless information" /;'  file

#OR

~$ raku -ne '.put unless my $useless and m:s/message repeated/ or $useless = m:s/useless information/;'  file 

Taking @Stéphane Chazelas' excellent Perl answer and translating to Raku. Note Raku is more explicit about what gets output. Here, .put is short for $_.put where $_ is the topic variable loaded with line-wise text. (Also, put adds the newline terminator for you).

The second difference is whitespace is not significant by default within regex matchers. So (from the first example), message \s repeated will match the input, as will the double-quoted string "message repeated".

But unquoted message repeated and/or unquoted useless information both will fail to match with the profuse error:

Space is not significant here; please use quotes or :s (:sigspace) modifier (or, to suppress this warning, omit the space, or otherwise change the spacing)

Hence the genesis of the second and third code examples, above.

https://docs.raku.org/language/regexes#Sigspace
https://docs.raku.org/routine/put
https://raku.org