Extract Strings from Web Log file

Question

Given a file containing web access logs for a YouTube video, every line is a hit and is in the format.

62.172.72.131 - - [02/Jan/2003:02:06:41 -0700] "GET /random/html/riaa_hacked/ HTTP/1.0" 200 10564 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; WWP 17 August 2001)"    
63.194.21.74 - - [30/Apr/2003:13:13:22 -0700] "GET /random/video/Star_Wars_Kid_Remix.wmv HTTP/1.1" 206 1146708 "-" "NSPlayer/9.0.0.2980 WMFSDK/9.0"    
161.114.88.73 - - [02/May/2003:03:27:41 -0700] "GET /random/video/Star_Wars_Kid.php HTTP/1.0" 302 1 "http://friends.portalofevil.com/sp.php?si=3&fi=FRIENDSOF&ti=1000489621&pi=1000489621" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; compaq)"    
64.164.63.70 - - [02/May/2003:13:24:19 -0700] "GET /random/video/Star_Wars_Kid.wmv HTTP/1.1" 302 307 "http://blogdex.media.mit.edu/" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)"

I need to extract the IP address and Date in the square bracket and put it in a csv file i.e ip_address,date

I'm using the following commands to get the IP and Date respectively

grep -oP '([0-9]{1,3}\.){3}[0-9]{1,3}' test.log
grep -oP "\[\K[^\]]+" test.log

I don't know how to combine it into 1 string to put it in a csv

tr '\n' > file.csv

Since this is a large log file, I thought unix commands will deal with it efficiently. Is there a difference between using unix commands or programming it in python (reading each line, manipulate the string and then write to file)?

Answer 1

You'd do better using awk,

awk '{print $1,$4,$5;}' test.log

awk breaks up each line on spacing, letting you refer to the fields as $1, $2,... etc, and then you just print the first, fourth and fifth fields ($4 and $5 makes up the date stamp).

Answer 2

Use RE substitutions in sed, here \1, \2, ... are assigned the values between corresponding \( and \)

sed 's/\([0-9\.]\) - - \[\(.*\)\] "GET .*/\1, \2/' test.log

(sure you may substitute a more exact patterns in the parentheses)

Answer 3

With a unix command you could use the following sed

sed -e 's/\(\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}\).*\[\(.*\)\].*/\1\t\3/' test.log

But if the logfile is very big, I think it would be better to use Python as it can handle actions on big files more effectively with e.g. the fileinput library or a generator.