1

Let me preface I am not sure if this question has been asked before, I have been Googling for answers but came up short.

I want to use standard Linux/Unix commands (running this on FreeBSD) to exclude lines from a log file that match a pattern. The log file also includes "last message repeated x times" to condense log entries.

As an example, I want to take this:

May 27 2023 11:07 relevant information #1
May 27 2023 11:07 relevant information #2
May 27 2023 11:08 last message repeated 3 times
May 27 2023 11:08 useless information #1
May 27 2023 11:08 last message repeated 5 times
May 27 2023 11:09 last message repeated 8 times
May 27 2023 11:09 relevant information #3
May 27 2023 11:09 useless information #2
May 27 2023 11:10 useless information #3
May 27 2023 11:10 last message repeated 6 times

And get this output:

May 27 2023 11:07 relevant information #1
May 27 2023 11:07 relevant information #2
May 27 2023 11:08 last message repeated 3 times
May 27 2023 11:09 relevant information #3

I've gotten as far as using sed commands to do this, but I don't know enough about how it works to figure it out. I am especially lost when it comes to the log lines that have multiple "last message repeated" following it. Here's what I'm working with currently:

sed '/useless information/{d;N;/last message repeated/d;}' ./logfile.txt

The above first deletes matching lines containing "useless information", then adds the next line to the namespace with N, and then is supposed to delete the resulting line if it contains "last message repeated". But it is only deleting the lines with "useless information".

4
  • Are the time stamps unique? Are the messages you want discarded all identical for each time stamp? Commented May 27, 2023 at 17:47
  • 1
    The messages I want to discard all contain the same string, "useless information" in my example. The messages I want to keep are any that do not contain "useless information". There can be multiple of the same message per timestamp, and there can be gaps in timestamps between "last message repeated" lines that follow.
    – ekrekeler
    Commented May 27, 2023 at 23:27
  • 2
    You can disable the collection of "repeated" lines by starting syslogd with the -c option specified twice. This would make the task trivial.
    – Kusalananda
    Commented May 28, 2023 at 8:49
  • 2
    Can you make your requirements positive by testing for and printing lines containing "relevant information" rather than negative by printing lines that do not contain "useless information"?
    – Ed Morton
    Commented May 29, 2023 at 0:02

3 Answers 3

1

Using any awk in any shell on every Unix box if you can test for relevant rather than useless information:

awk '/last message repeated/ && f; {f=/relevant information/} f' file
May 27 2023 11:07 relevant information #1
May 27 2023 11:07 relevant information #2
May 27 2023 11:08 last message repeated 3 times
May 27 2023 11:09 relevant information #3
0

With pcregrep:

$ pcregrep -vM 'useless information(.*\n.*message repeated)*' your-file
May 27 2023 11:07 relevant information #1
May 27 2023 11:07 relevant information #2
May 27 2023 11:08 last message repeated 3 times
May 27 2023 11:09 relevant information #3

-M is pcregrep's Multiline mode which pulls extra lines into the pattern space as needed for the regexp to match (within limits). With that enabled the m perl/PCRE flag (which makes ^ and $ match at the beginning/end of each line within the subject as opposed to just the beginning/end of the subject) is enabled and the s flag (which makes . also match newline characters) is not enabled which explains why those .*s above don't gobble up the whole input despite being greedy.

With sed, that could be for instance:

sed -n '
  :start
  /useless information/ {
    :more
    n
    /message repeated/ b more
    b start
  }
  p'

With awk:

awk '! (useless && /message repeated/ || \
        (useless = /useless information/))'

Note that (...) are needed around the assignment to useless in busybox awk at least.

Same in perl:

perl -ne 'print unless $useless and /message repeated/ or
            $useless = /useless information/'

(using and/or instead of &&/|| as they have lower precedence than = removing the need for parenthesis; see perldoc perlop for reference).

1
  • Thank you for the solution! I didn't know pcregrep existed, I find this solution to be the most elegant. The awk and perl ones are great as well since you can find them on most systems.
    – ekrekeler
    Commented May 28, 2023 at 15:17
0

Using Raku (formerly known as Perl_6)

~$ raku -ne '.put unless my $useless and /message \s repeated/ or $useless = /useless \s information/;'  file

#OR

~$ raku -ne '.put unless my $useless and / "message repeated" / or $useless = / "useless information" /;'  file

#OR

~$ raku -ne '.put unless my $useless and m:s/message repeated/ or $useless = m:s/useless information/;'  file 

Taking @Stéphane Chazelas' excellent Perl answer and translating to Raku. Note Raku is more explicit about what gets output. Here, .put is short for $_.put where $_ is the topic variable loaded with line-wise text. (Also, put adds the newline terminator for you).

The second difference is whitespace is not significant by default within regex matchers. So (from the first example), message \s repeated will match the input, as will the double-quoted string "message repeated".

But unquoted message repeated and/or unquoted useless information both will fail to match with the profuse error:

Space is not significant here; please use quotes or :s (:sigspace) modifier (or, to suppress this warning, omit the space, or otherwise change the spacing)

Hence the genesis of the second and third code examples, above.

https://docs.raku.org/language/regexes#Sigspace
https://docs.raku.org/routine/put
https://raku.org

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .