6

I want to filter a log file and print some lines between two matches and print only the last occurrence.

Sample file content:

2023-03-08 11:12:44,306 - Code Deploy - INFO - Received signal
2023-03-08 11:12:44,306 - Code Deploy - INFO - Received message signal
2023-03-08 11:12:44,306 - Code Deploy - INFO - Branch is Testing
2023-03-08 11:12:44,307 - Code Deploy - INFO - Deployment started
2023-03-08 11:13:31,782 - Code Deploy - INFO - Old version2_0_5_12
2023-03-08 11:13:31,783 - Code Deploy - INFO - New version2_0_5_13
2023-03-08 11:13:32,553 - Code Deploy - INFO - Permission fixed
2023-03-08 11:13:32,554 - Code Deploy - INFO - Deployment finished
2023-03-08 11:13:34,900 - Code Deploy - ERROR - !!!!!!!!!! EXCEPTION !!!!!!!!!(535, b'5.7.8     Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials z16-20020a170903019000b0019a97a4324dsm9818181plg.5 - gsmtp')Traceback (most recent call last):
File "/root/code-dployment/server/deploy.py", line 94, in send_email
server.login(gmail_user, gmail_password)
File "/usr/lib/python3.5/smtplib.py", line 729, in login
raise last_exception
File "/usr/lib/python3.5/smtplib.py", line 720, in login
initial_response_ok=initial_response_ok)
File "/usr/lib/python3.5/smtplib.py", line 641, in auth
raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (535, b'5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials z16-20020a170903019000b0019a97a4324dsm9818181plg.5 - gsmtp')

2023-03-09 11:52:57,194 - Code Deploy - INFO - Received signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received message signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Branch is Testing
2023-03-09 11:52:57,195 - Code Deploy - INFO - Deployment started
2023-03-09 11:53:58,246 - Code Deploy - INFO - Old version2_0_5_13
2023-03-09 11:53:58,246 - Code Deploy - INFO - New version2_0_5_14
2023-03-09 11:53:58,498 - Code Deploy - INFO - Permission fixed
2023-03-09 11:53:58,498 - Code Deploy - INFO - Deployment finished
2023-03-09 11:54:00,797 - Code Deploy - ERROR - !!!!!!!!!! EXCEPTION !!!!!!!!!(535, b'5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials k17-20020aa790d1000000b005907716bf8bsm11097506pfk.60 - gsmtp')Traceback (most recent call last):
File "/root/code-dployment/server/deploy.py", line 94, in send_email
server.login(gmail_user, gmail_password)
File "/usr/lib/python3.5/smtplib.py", line 729, in login
raise last_exception
File "/usr/lib/python3.5/smtplib.py", line 720, in login
initial_response_ok=initial_response_ok)
File "/usr/lib/python3.5/smtplib.py", line 641, in auth
raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (535, b'5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials k17-20020aa790d1000000b005907716bf8bsm11097506pfk.60 - gsmtp')

It needs to get the contents between two patterns.

Pattern1 = 'Received signal'

Pattern2 = 'Deployment finished'

Expected result:

2023-03-09 11:52:57,194 - Code Deploy - INFO - Received signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received message signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Branch is Testing
2023-03-09 11:52:57,195 - Code Deploy - INFO - Deployment started
2023-03-09 11:53:58,246 - Code Deploy - INFO - Old version2_0_5_13
2023-03-09 11:53:58,246 - Code Deploy - INFO - New version2_0_5_14
2023-03-09 11:53:58,498 - Code Deploy - INFO - Permission fixed
2023-03-09 11:53:58,498 - Code Deploy - INFO - Deployment finished

I want to get an AWK command to use it in a bash script. I found a solution to filter the contents between two patterns by using the command below:

# awk '/Received signal/,/Deployment finished/' /tmp/result.log

It will print the whole matching lines of all occurrences, but, I need to filter it so it only prints the last occurrence of the matching pattern.

The above command's output is:

2023-03-08 11:12:44,306 - Code Deploy - INFO - Received signal
2023-03-08 11:12:44,306 - Code Deploy - INFO - Received message signal
2023-03-08 11:12:44,306 - Code Deploy - INFO - Branch is Testing
2023-03-08 11:12:44,307 - Code Deploy - INFO - Deployment started
2023-03-08 11:13:31,782 - Code Deploy - INFO - Old version2_0_5_12
2023-03-08 11:13:31,783 - Code Deploy - INFO - New version2_0_5_13
2023-03-08 11:13:32,553 - Code Deploy - INFO - Permission fixed
2023-03-08 11:13:32,554 - Code Deploy - INFO - Deployment finished
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received message signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Branch is Testing
2023-03-09 11:52:57,195 - Code Deploy - INFO - Deployment started
2023-03-09 11:53:58,246 - Code Deploy - INFO - Old version2_0_5_13
2023-03-09 11:53:58,246 - Code Deploy - INFO - New version2_0_5_14
2023-03-09 11:53:58,498 - Code Deploy - INFO - Permission fixed
2023-03-09 11:53:58,498 - Code Deploy - INFO - Deployment finished

4 Answers 4

5

Using any awk and very similar to one of the scripts in @terdon's answer but IMO just a bit more idiomatically awkish in use of awk's condition { action } body structure:

$ awk '
    /Received signal/ { f=1; rec="" }
    f { rec = rec $0 ORS }
    /Deployment finished/ { f=0 }
    END { if (f=="0") printf "%s", rec }
' file
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received message signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Branch is Testing
2023-03-09 11:52:57,195 - Code Deploy - INFO - Deployment started
2023-03-09 11:53:58,246 - Code Deploy - INFO - Old version2_0_5_13
2023-03-09 11:53:58,246 - Code Deploy - INFO - New version2_0_5_14
2023-03-09 11:53:58,498 - Code Deploy - INFO - Permission fixed
2023-03-09 11:53:58,498 - Code Deploy - INFO - Deployment finished

The minor functional differences between this and @terdon's answer are:

  1. This one will produce the desired record terminators in the unlikely event that you decide to set ORS to something different from RS (e.g. maybe you want to convert RS='\r\n' to ORS='\n'), while @terdon's will reproduce the RS value in most of the output with an ORS just at the end of it.
  2. @terdon's would print a blank line if the input file doesn't contain any Received signal lines while this one wouldn't produce any output.
  3. This one will only print the text between the delimiters if both delimiters exist in the input while @terdon's would print whatever follows a Received signal line even if no subsequent Deployment finished line is present.

Regarding awk '/Received signal/,/Deployment finished/' /tmp/result.log in your question - don't use range expressions, use a flag, see is-a-start-end-range-expression-ever-useful-in-awk. As you can see in every answer posted so far that uses a range expression, it requires testing the same condition twice.

2
  • 1
    Heh, and I only even used v=v RS $0 instead of just v=v" "$0 thinking that you would come and correctly point out that for all I know, the RS wasn't a space! For what it's worth, this is better than mine in my opinion as well.
    – terdon
    Commented Mar 10, 2023 at 18:43
  • 1
    @terdon Yeah, it was the use of RS instead of ORS when building the output record that got my eye twitching a bit on that one :-). I'd just have commented but I wanted to showcase more the condition { action } structure anyway.
    – Ed Morton
    Commented Mar 10, 2023 at 20:00
4

Reverse the file, delete all lines not in the range /Deployment finished/,/Received signal/, and quit as soon as you match the end of that range of lines. Then reverse the result.

tail -r file |
sed -e '/Deployment finished/,/Received signal/!d' -e '/Received signal/q' |
tail -r

GNU system users may want to use tac in place of tail -r above.

If you feel you need to use awk instead, replace the sed command in the pipeline above with

awk '/Deployment finished/,/Received signal/; /Received signal/ { exit }'

Output, given the data in the question:

2023-03-09 11:52:57,194 - Code Deploy - INFO - Received signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received message signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Branch is Testing
2023-03-09 11:52:57,195 - Code Deploy - INFO - Deployment started
2023-03-09 11:53:58,246 - Code Deploy - INFO - Old version2_0_5_13
2023-03-09 11:53:58,246 - Code Deploy - INFO - New version2_0_5_14
2023-03-09 11:53:58,498 - Code Deploy - INFO - Permission fixed
2023-03-09 11:53:58,498 - Code Deploy - INFO - Deployment finished
3
  • Thank you, It is a nice idea. I think you have a typo in the result, you mean the command "tac" not "tail". >>> # tac file | sed -e '/Deployment finished/,/Received signal/!d' -e '/Received signal/q' | tac Commented Mar 10, 2023 at 10:25
  • 2
    @AmalPRamesh As you can see just after the initial command, I suggest that users that use GNU systems change tail -r to tac. The tail -r command works the same way on macOS and other BSD systems as GNU tac on GNU systems. As the user in the question did not say they were on any particular Unix, I made sure to mention both tac and tail -r.
    – Kusalananda
    Commented Mar 10, 2023 at 10:28
  • 1
    @AmalPRamesh Please read. Commented Mar 10, 2023 at 10:35
3

A simple trick is to save each match in a variable, overwriting what was previously there, and then, at the end of the script, print the variable:

$ awk '{ 
         if(/Received signal/){k=1; v=$0} 
         else if(k==1){
           v=v RS $0; 
           if(/Deployment finished/){ k=0 }
         }
       } 
       END{ print v }' result.log
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received message signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Branch is Testing
2023-03-09 11:52:57,195 - Code Deploy - INFO - Deployment started
2023-03-09 11:53:58,246 - Code Deploy - INFO - Old version2_0_5_13
2023-03-09 11:53:58,246 - Code Deploy - INFO - New version2_0_5_14
2023-03-09 11:53:58,498 - Code Deploy - INFO - Permission fixed
2023-03-09 11:53:58,498 - Code Deploy - INFO - Deployment finished

Or you could use tac to reverse the file and then print the first match:

$ tac result.log | 
   awk '/Deployment finished/,/Received signal/{
      print; 
      if(/Received signal/){ exit }
   }' | tac
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received message signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Branch is Testing
2023-03-09 11:52:57,195 - Code Deploy - INFO - Deployment started
2023-03-09 11:53:58,246 - Code Deploy - INFO - Old version2_0_5_13
2023-03-09 11:53:58,246 - Code Deploy - INFO - New version2_0_5_14
2023-03-09 11:53:58,498 - Code Deploy - INFO - Permission fixed
2023-03-09 11:53:58,498 - Code Deploy - INFO - Deployment finished
0

Using Raku (formerly known as Perl_6)

~$ raku -e 'my $k; my @v; for lines() { 
            if /Received \s signal/ {$k = 1; @v = $_} 
            elsif ($k == 1) { @v.push: $_ }; 
            if /Deployment \s finished/ {$k = 0}
            }; .put for @v;'  file
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Received message signal
2023-03-09 11:52:57,194 - Code Deploy - INFO - Branch is Testing
2023-03-09 11:52:57,195 - Code Deploy - INFO - Deployment started
2023-03-09 11:53:58,246 - Code Deploy - INFO - Old version2_0_5_13
2023-03-09 11:53:58,246 - Code Deploy - INFO - New version2_0_5_14
2023-03-09 11:53:58,498 - Code Deploy - INFO - Permission fixed
2023-03-09 11:53:58,498 - Code Deploy - INFO - Deployment finished

The above code shamelessly follows @terdon's excellent awk code, but re-written in Raku. The key as (@terdon notes) is to over-write the "storing-variable" (in this case @v array) every time the opening /Received \s signal/ regex is spotted. Here, because lines are loaded one-by-one into the $_ topic variable, the code @v = $_ is used.

If--upon reaching the end of the file--you don't want any return unless the closing /Deployment \s finished/ is seen last (i.e. a "full-record" requirement), then change the final output statement to:

.put if $k == 0 for @v;

#OR

.put unless $k == 1 for @v;

https://docs.raku.org/language/control.html#Control_flow
https://raku.org

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .