0

Here is the sample file. The actual log is way bigger than this.

user@linux:~$ cat log.txt 
[24/09/2018:22:41:49 GMT] "PROXY_SERVER_BA2" c-ip=x.x.x.x cs-bytes=1198 cs-categories="Technology/Internet" cs-host=shavar.services.mozilla.com cs-ip=y.y.y.y cs-method=CONNECT cs-uri-path=/ cs-uri-port=443 cs-uri-scheme=tcp cs-User-Agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" cs-username=johndoe dnslookup-time=0 duration=60 rs-status=0 s-action=TCP_TUNNELED

[24/09/2018:17:45:44 GMT] "PROXY_SERVER_AA2" c-ip=x.x.x.x cs-bytes=152450 cs-categories="Business/Economy" cs-host=cvshipping.ups.com cs-ip=z.z.z.z cs-method=CONNECT cs-uri-path=/ cs-uri-port=443 cs-uri-scheme=tcp cs-User-Agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" cs-username=johndoe dnslookup-time=0 duration=268 rs-status=0 s-action=TCP_TUNNELED

[24/09/2018:17:44:03 GMT] "PROXY_SERVER_AA2" c-ip=x.x.x.x cs-bytes=1795 cs-categories="Software Downloads" cs-host=blocklist.addons.mozilla.org cs-ip=z.z.z.z cs-method=CONNECT cs-uri-path=/ cs-uri-port=443 cs-uri-scheme=tcp cs-User-Agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" cs-username=johndoe dnslookup-time=0 duration=61 rs-status=0 s-action=TCP_TUNNELED

[24/09/2018:17:41:44 GMT] "PROXY_SERVER_AA2" c-ip=x.x.x.x cs-bytes=3882 cs-categories="Web Ads/Analytics" cs-host=cebwa.d2.sc.omtrdc.net cs-ip=z.z.z.z cs-method=CONNECT cs-uri-path=/ cs-uri-port=443 cs-uri-scheme=tcp cs-User-Agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" cs-username=johndoe dnslookup-time=0 duration=35 rs-status=0 s-action=TCP_TUNNELED

[20/09/2018:15:48:50 GMT] "PROXY_SERVER_AA2" c-ip=a.a.a.a cs-auth-group=Domain%20Users cs-bytes=227 cs-categories="Web Ads/Analytics;Suspicious" cs-host=data35.adlooxtracking.com cs-ip=b.b.b.b cs-method=CONNECT cs-uri-path=/ cs-uri-port=443 cs-uri-scheme=tcp cs-User-Agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0" cs-username=johndoe duration=0 rs-status=0 s-action=TCP_TUNNELED

[20/09/2018:15:48:35 GMT] "PROXY_SERVER_AA2" c-ip=a.a.a.a cs-auth-group=Domain%20Users cs-bytes=3201 cs-categories="Search Engines/Portals" cs-host=www.google.com cs-ip=b.b.b.b cs-method=CONNECT cs-uri-path=/ cs-uri-port=443 cs-uri-scheme=tcp cs-User-Agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0" cs-username=johndoe dnslookup-time=0 duration=117 rs-status=0 s-action=TCP_TUNNELED
user@linux:~$ 

My goal is to get timestamp, category, and the hostname for each line and redirect it to a new file.

timestamp (inside [ ])

user@linux:~$ sed -n 's/^.*\[//p' log.txt | cut -d ']' -f1
24/09/2018:22:41:49 GMT
24/09/2018:17:45:44 GMT
24/09/2018:17:44:03 GMT
24/09/2018:17:41:44 GMT
20/09/2018:15:48:50 GMT
20/09/2018:15:48:35 GMT
user@linux:~$ 

hostname (after cs-host=)

user@linux:~$ sed -n 's/^.*cs-host=//p' log.txt | cut -d\  -f1
shavar.services.mozilla.com
cvshipping.ups.com
blocklist.addons.mozilla.org
cebwa.d2.sc.omtrdc.net
data35.adlooxtracking.com
www.google.com
user@linux:~$ 

category (betweencs-categories=" and ")

user@linux:~$ sed -n 's/^.*cs-categories="//p' log.txt | cut -d '"' -f1
Technology/Internet
Business/Economy
Software Downloads
Web Ads/Analytics
Web Ads/Analytics;Suspicious
Search Engines/Portals
user@linux:~$ 

Instead of executing sed command one by one, is it possible to run a single command to produce an output like this?

Expected Output

24/09/2018:22:41:49 GMT    shavar.services.mozilla.com      Technology/Internet
24/09/2018:17:45:44 GMT    cvshipping.ups.com               Business/Economy
24/09/2018:17:44:03 GMT    blocklist.addons.mozilla.org     Software Downloads
24/09/2018:17:41:44 GMT    cebwa.d2.sc.omtrdc.net           Web Ads/Analytics
20/09/2018:15:48:50 GMT    data35.adlooxtracking.com        Web Ads/Analytics;Suspicious
20/09/2018:15:48:35 GMT    www.google.com                   Search Engines/Portals

Also, if you have better solution for this, please let me know.

2

How far would

sed -r '/^$/d; s/^[^[]*[[]([^]]*)[]].*cs-categories="([^"]*)".*cs-host=([^ ]*) .*/\1\t\3\t\2/' file
24/09/2018:22:41:49 GMT shavar.services.mozilla.com Technology/Internet
24/09/2018:17:45:44 GMT cvshipping.ups.com  Business/Economy
24/09/2018:17:44:03 GMT blocklist.addons.mozilla.org    Software Downloads
24/09/2018:17:41:44 GMT cebwa.d2.sc.omtrdc.net  Web Ads/Analytics
20/09/2018:15:48:50 GMT data35.adlooxtracking.com   Web Ads/Analytics;Suspicious
20/09/2018:15:48:35 GMT www.google.com  Search Engines/Portals

get you?

sed -r '                        use extended regular expressions in the script
/^$/d                           delete empty lines

s/^[^[]*[[]([^]]*)[]].*         look for date time string between square brackets and prepare for 
                                the first "back reference"
cs-categories="([^"]*)".*       look for the string after cs-categories and prepare for second "b r"
cs-host=([^ ]*)                 look for the string after cs-host and prepare for third "b r"
.*/\1\t\3\t\2/                  create output line from back references separated by <TAB> chars.
'
2

Using perl with look-aheads: this way it doesn't matter if the host comes before or after the categories

perl -lne '
    m(
        ^\[ (.*?) \]                   # match the timestamp
        (?=.* cs-categories= "(.+?)")  # look ahead for the category
        (?=.* cs-host= (\S+) )         # look ahead for the host
    )x
    and print join ",", $1,$2,$3
' log.txt 
  • Thanks @glenn. The perl code looks better and easier to read but it doesn't really work with sample data above. – Sabrina Sep 30 '18 at 3:50
  • It doesn't provide exactly the output you were looking for, but yes it does work with that data. What do you think I tested with? – glenn jackman Sep 30 '18 at 11:22
0

How about using awk:

awk '{t=""; h=""; c=""; for (i=1; i<=NF; i++) {if ($i ~ /^\[/) {t=$i} if ($i ~/^cs-host=/) {h=$i} if ($i ~ /^cs-categories=/) {c=$i}} if ((t != "") && (h != "") && (c != "")) printf("%s %s %s\n", t, h, c)}' _inputfile_

This is a rough estimate of what you could do. Basically it just loops through each whitespace field in a given line and checks if the line starts with a certain string. If then puts the value of that field into a certain variable. After all of the fields have been processed, if all 3 fields are not blank, it prints out the fields. It then moves onto the next line in the input file.

I did not do anything to deal with white space in the string. You could do further checks within one of the existing checks to see if the string ends with a double-quote. If not, append the next field to the variable.

I also did not do anything with substrings to get rid of things like [, ] and ". I leave that as an exercise for you. :)

  • 1) unreadable, add some newlines to your code. 2) wrong results (category values can contain spaces) – glenn jackman Sep 26 '18 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.