Parse a log using perl

Question

Im looking to parse a log using a perl command that I designed. So far all I would like to accomplish is pulling a specific timestamp format and retrieve the host. the log would look like this

2016-05-07T09:07:04.933343+00:00 heroku[router]: status=301 bytes=680 service=2698ms connect=1ms dyno=web.2 fwd="10.29.10.29" at=info host="jamaican.com" request_id=32fc8d88-99f8-4cc2-89f9-284d059eebf8 method=GET path="/blog"

my command so can parse the date format I want but I cant seem to figure out how to add the host in the output. any recommendations would be great!

cat test.log  |
perl -lne 'print $1 if  /^([0-9]+[-]+[0-9]+[0-9]+[-]+[0-9]+[0-9]+[T]+[0-9]+[0-9]+[:]+[0-9]+[0-9]+[:]+[0-9]+[0-9])/'

Answer 1

Try this:

perl -nle'($time, $host) = /^(\S+)\s(?:\S+\s+){8}\S+="(\S+?)"/; print "$time $host"'

Output:

2016-05-07T09:07:04.933343+00:00 jamaican.com

• \S means non-space
• \s is space
• (?:) is a logical grouping that's not captured
• {8} are the skipped "words"
• \S+="(\S+?)" means: skip until = and capture what's between the two " quotes
• ($time, $host) = /.../ assigns the two captured groups to $time and $host

Answer 2

You need to capture the hostname with a second capture group.

e.g. using your sample input:

$ cat test.log 
2016-05-07T09:07:04.933343+00:00 heroku[router]: status=301 bytes=680 service=2698ms connect=1ms dyno=web.2 fwd="10.29.10.29" at=info host="jamaican.com" request_id=32fc8d88-99f8-4cc2-89f9-284d059eebf8 method=GET path="/blog"

This perl one-liner extracts both timestamp and the hostname field. I've improved the regex a little as well, using \d for digits, with a match count for each.

$ perl -lne 'print "$1 $2" if (m/(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+\+\d{2}:\d{2}) ([^ ]+) /)' test.log
2016-05-07T09:07:04.933343+00:00 heroku[router]:

Another alternative:

$ perl -lne 'print "$1 $2 $3" if (m/(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+\+\d{2}:\d{2}) ([^[]+)\[([^]]+)\]/)' test.log
2016-05-07T09:07:04.933343+00:00 heroku router

I assumed you wanted hostname of the local machine (I didn't even notice the host="jamaican.com" part of the log entry). That's probably not what you wanted, so if you want the hostname that's inside the double-quotes after host=, then:

$ perl -lne 'print "$1 $2" if (m/(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+\+\d{2}:\d{2}).*host="([^"]+)"/)' test.log
2016-05-07T09:07:04.933343+00:00 jamaican.com 

or (much simpler):

$ perl -lne 'print "$1 $2" if (m/(^\S+).*host="([^"]+)"/)' test.log
2016-05-07T09:07:04.933343+00:00 jamaican.com

Or extract the timestamp, parse it with Date::Parse and reformat it with Date::Format.

$ perl -MDate::Parse -MDate::Format -lne \
  'if (m/(^\S+).*host="([^"]+)"/) {
     print join(" ", time2str("%Y-%m-%d %R %z",str2time($1)), $2)
   }' test.log
2016-05-07 19:07 +1000 jamaican.com

Note: the timestamp has been converted to local timezone (for me, that's +1000 or Australian Eastern Standard Time). time2str() uses local timezone by default, but you can give it a third argument (time2str(TEMPLATE, TIME [, ZONE])) to make it output the time in any other zone.

$ perl -MDate::Parse -MDate::Format -lne   'if (m/(^\S+).*host="([^"]+)"/) {
     print join(" ", time2str("%Y-%m-%d %R %z",str2time($1),"+0"), $2)
   }' test.log
2016-05-07 09:07 +0000 jamaican.com