2

I have a log file that look like:

2016-05-31 09:54:36 (16667) heritage_w?
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?i=290
  #accesses 3,435 (#welcome 415) since 03/07/2012
2016-05-31 09:54:41 (16677) heritage_w?w=
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?
  #accesses 3,436 (#welcome 416) since 03/07/2012
2016-06-01 04:07:06 (22190) heritage_w?m=MOD_IND;i=88
  From: ubunzeus
  User: user2 (wizard)
  Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?i=88
  #accesses 3,623 (#welcome 441) since 03/07/2012    
2016-06-01 04:07:38 (22255) heritage_w?m=MOD_FAM;i=28;ip=88
  From: ubunzeus
  User: user2 (wizard)
  Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?m=MOD_IND;i=88
  #accesses 3,624 (#welcome 441) since 03/07/2012

I'm trying to make the undented lines as the Record Separator RS.

Using code similar to:

$ gawk 'BEGIN{RS="^2016"}; /user1/ {print}'

I would hope to a printout of only the records that have "user1" in them.

Currently the command line is printing the whole file... all the records.

This is the expected output:

2016-05-31 09:54:36 (16667) heritage_w?
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?i=290
  #accesses 3,435 (#welcome 415) since 03/07/2012
2016-05-31 09:54:41 (16677) heritage_w?w=
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?
  #accesses 3,436 (#welcome 416) since 03/07/2012

Clarifications of the specifics of this question

I accepted the answer John1024 which provides for me to be able to pick out the needed records. However, I hope someone will eventually have input on how to use the specific regex feature as the Record Separator (RS) variable, which in this case would be the unindented line.

I took the string of what I was using as described by John1024 and used the non-white regex expression in various combination, but it doesn't work.

The lines that I use that doesn't properly filter the records are:

$ gawk 'BEGIN{RS='\n\S'}; /user1/ {print}' event.log
$ gawk 'BEGIN{RS='\S'}; /user1/ {print}' event.log
$ gawk 'BEGIN{RS="\n^\S"}; /user1/ {print}' event.log
$ gawk 'BEGIN{RS="^\S"}; /user1/ {print}' event.log

All the above combinations displays all the records. I'm sure the single quotes '^\S' is using the actual charecters and not the escaped meaning. The double quoted ones "^\S" are giving the error message:

gawk: cmd. line:1: warning: escape sequence `\S' treated as plain `S'

I'm able to verify that "\S" will regex non-white first column characters this. It shows online the un-indented lines:

$ egrep "^\S" event.log

Output of the above cli:

2016-05-31 09:54:36 (16667) heritage_w?
2016-05-31 09:54:41 (16677) heritage_w?w=
2016-06-01 04:07:06 (22190) heritage_w?m=MOD_IND;i=88
2016-06-01 04:07:38 (22255) heritage_w?m=MOD_FAM;i=28;ip=88

With the help of the accepted answer... the newline code and addressing the escape character error by using a double backslash, the following filters the desired records:

$ gawk 'BEGIN{RS="\n\\S"}; /user1/ {print}' event.log
  • In a completely different direction that you may already be aware of, you could use something like logstash (why not use the ELK stack?) to parse these records out. – Wayne Werner Jun 9 '16 at 12:16
2

Try:

$ gawk 'BEGIN{RS="\n2016"}; /user1/ {print}' input

This produces the output;

2016-05-31 09:54:36 (16667) heritage_w?
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?i=290
  #accesses 3,435 (#welcome 415) since 03/07/2012
-05-31 09:54:41 (16677) heritage_w?w=
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?
  #accesses 3,436 (#welcome 416) since 03/07/2012

Notice that the second record is missing the initial 2016. That is. of course, because that 2016 became part of the record separator. If you want to restore that part before beginning any processing of a record:

gawk 'BEGIN{RS="\n2016"} NR>1{$0="2016" $0;} /user1/ {print}' input

Improvement

This version restores text as needed to the beginning of each line:

gawk '{$0=substr(last,2)$0;} /user1/{print} {last=RT}' RS='\n[^[:space:]]' input

How it works:

  • {$0=substr(last,2)$0;} prepends to $0 the text that had been removed by the record separator. substr is used to remove the preceding newline.

  • /user1/{print} prints the records that we are interested in.

  • {last=RT} saves the actual record separator so that part of it can be prepended to the next record. RT is a GNU extension and is not supported by other versions of awk.

  • RS='\n[^[:space:]]' sets the record separator to a newline followed by any non-space. Using a regex as a record separator works with GNU awk.

Example:

$ gawk '{$0=substr(last,2)$0;} /user1/{print} {last=RT}' RS='\n[^[:space:]]' input
2016-05-31 09:54:36 (16667) heritage_w?
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?i=290
  #accesses 3,435 (#welcome 415) since 03/07/2012
2016-05-31 09:54:41 (16677) heritage_w?w=
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?
  #accesses 3,436 (#welcome 416) since 03/07/2012
  • Thanks. it works great. I'm just wondering, is there a way to retain the missing string (which is the record separator)? Also, is there a way to make the un-indented line actually be the record separator without specifying the (2016) string? – L. D. James Jun 1 '16 at 17:10
  • With gawk, you could use the RT variable to restore the actual matching separator, and then use a RE for RS e.g. gawk 'BEGIN{RS="\n[^ \t]";ORS=""}; /user1/ {print $0,RT}'. See examples at Record Splitting with gawk – steeldriver Jun 1 '16 at 17:56
  • @steeldriver: Interesting concept, but it doesn't quite work. If we select /user1/ an extra 2 (with no trailing newline) is printed after the desired data; If we select /user2/ the initial 2 is omitted and an extra space is printed, (once again, with no trailing newline). – PM 2Ring Jun 1 '16 at 21:07
  • Hi, John... you might consider adding $ gawk 'BEGIN{RS="\n\\S"}; /user1/ {print}' event.log to your formatted answer. The details are provided in the question. Thanks again for the invaluable help! – L. D. James Jun 2 '16 at 7:59
  • @L.D.James Very good. I modified that somewhat so that the code restores the character that was captured by the \\S. See the updated answer. By the way, \\S is GNU only. That is fine here because we are using other GNU-only features. For the record, though, a more portable regex for expressing non-space is [^[:space:]]. – John1024 Jun 2 '16 at 21:21
2

Here's a slightly different strategy. We accumulate each indented line into a holding buffer. When an un-indented line is read we call a function that prints the buffer if it contains the desired pattern and then replaces the buffer contents with the new header line. We also need to call that function when the end of the file is reached.

#!/usr/bin/awk -f
#   Select records from a file 
#   Each record header line is unindented and each record body line is indented
#   Written by PM 2Ring 2015.06.02

function ShowSelected()
{
    if (hold ~ /User: user1/)
        printf "%s", hold
    hold = $0 ORS
}

/^ /{hold = hold $0 ORS; next}

{ShowSelected()}

END{ShowSelected()}

Here's a one-line version:

awk 'function S(){if(h~/User: user1/)printf "%s",h; h=$0 ORS}; /^ /{h=h $0 ORS; next}; {S()};END{S()}'

Just for fun, here's a sed version. It uses essentially the same algorithm.

sed '/^ /!bA;H;$bA;d;:A;x;/User: user1/!d'

Here's the same thing, with comments.

#!/bin/sed -f    
#   Select records from a file 
#   Each record header line is unindented and each record body line is indented
#   Written by PM 2Ring 2015.06.02

# If line doesn't start with a space, branch to the select & display routine
/^ /!bA

# Append pattern space (i.e., the current line) to the hold space
H

# If this is the last line, branch to the select & display routine
$bA

# Delete the pattern space and start the next cycle
d

# The select & display routine
:A

# Exchange the contents of the hold and pattern spaces
x

# Delete the pattern if it doesn't contain the regex /User: user1/
# if the pattern isn't deleted it will be printed
/User: user1/!d

Here's a sed - awk hybrid approach, inspired by Thor's idea of using sed to do some pre-processing. We prefix each un-indented line with a \xff character, and then use that as the awk record separator. This won't work correctly if the log file uses that \xff character itself, but hopefully that will not be the case. :)

<logfile sed 's/^[^ ]/\xff&/' | awk 'BEGIN{RS="\xff";ORS=""};/User: user1/'
1

I would preprocess the file with e.g. sed. So to extract the second line of each record, do something like this:

<infile sed 's/^[^ ]/&\n/' | awk '{ print $2 }' RS= FS='\n'

Output:

  From: ip68-8-49-100.sd.sd.cox.net
  From: ip68-8-49-100.sd.sd.cox.net
  From: ubunzeus
  From: ubunzeus

Edit - How to print every record where $3 contains user1:

<infile sed '1!s/^[^ ]/\n&/' | awk '$3 ~ /user1/' RS= FS='\n'

Output:

2016-05-31 09:54:36 (16667) heritage_w?                                
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?i=290
  #accesses 3,435 (#welcome 415) since 03/07/2012
2016-05-31 09:54:41 (16677) heritage_w?w=
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?
  #accesses 3,436 (#welcome 416) since 03/07/2012
  • 1
    Thanks for the input. But I need all the information from the all records that meets the search criteria. – L. D. James Jun 1 '16 at 17:32
  • @L.D.James: This is the cleanest approach, printing the From line was just an example. In order to replicate what you have added to your question, see my edit. – Thor Jun 2 '16 at 8:38
  • Thanks. These are some of the things that I put in the /search/ criteria. !/ubunzeus/, /ubunzeus/, /cox.net/. Then I'm looking to review elements of the log, I just change the file before the {print} parameter. The awk one liner does everything needed to complement my other scripts using awk. I was just having problems using the regex unindented line as the Record Separator. The \n along with the double slash for an escape character fixed everything. I'll explore the ORS variable to get the Record Separator character(s) back. – L. D. James Jun 2 '16 at 8:56
0

IMO, the easiest way is to use sed to transform the input to paragraph-separated records (one or more blank lines between each record). In other words, skipping the very first line, insert a newline before every line that doesn't begin with a blank (space or tab).

Then you can just tell awk to use two or more newlines as the input record separator (RS) with RS='\n\n+'.

BTW, there's no need to set the output record separator (ORS) to be the same unless you want the output to also be in paragraphs. You didn't ask for that, so I didn't include it. If that's what you want (e.g. because you want to do some further processing on the output) then add -v ORS='\n\n' to the awk options.

$ sed -e '2,$ s/^[^[:blank:]]/\n&/' ldjames.txt | 
    awk -v RS='\n\n+' '/user1/ {print}'
2016-05-31 09:54:36 (16667) heritage_w?
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?i=290
  #accesses 3,435 (#welcome 415) since 03/07/2012
2016-05-31 09:54:41 (16677) heritage_w?w=
  From: ip68-8-49-100.sd.sd.cox.net
  User: user1wizard (wizard)
  Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
  Referer: http://dbase.apollo3.com/heritage_w?
  #accesses 3,436 (#welcome 416) since 03/07/2012

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.