2

We have a log file that I often follow live with tail and use grep to filter for the lines I'm interested in. However, the lines contain a lot of data I'm not always interested in but they have been difficult for me to parse so I only see the portions of the line that I want. The format of each line entry is primarily a list tags and the data (sometimes containing spaces) surrounded by quotes. Here are some sample (sanitized) log lines:

2017:11:29-11:29:56 filter-1 httpproxy[3194]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="10.11.12.13" dstip="14.3.1.4" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_HttPro1234 (Campus2)" filteraction="REF_HttStu (Allow Policy)" size="6518" request="0x915a3e00" url="https://website.net/" referer="" error="" authtime="0" dnstime="1" cattime="73" avscantime="0" fullreqtime="61576999" device="0" auth="6" ua="" exceptions="" category="9998" reputation="unverified" categoryname="Uncategorized" country="United States" application="krux" app-id="826"
2017:11:29-11:29:56 filter-1 httpproxy[3194]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="GET" srcip="10.13.14.15" dstip="154.6.75.10" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_HttPro1235 (Campus1)" filteraction="REF_HttStu (Allow Policy)" size="3161" request="0x6b4d5610" url="http://host.com/mini_banner.png" referer="http://www.web.com/computers.htm" error="" authtime="0" dnstime="0" cattime="64" avscantime="848" fullreqtime="50046" device="0" auth="6" ua="Mozilla/5.0 (X11; CrOS x86_64 9765.85.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.123 Safari/537.36" exceptions="" category="111" reputation="trusted" categoryname="Education/Reference" sandbox="-" content-type="image/png"

One thing to note is that all the tags are not present on every line. For example, application and app-id are present in the first line but not the second.

Using the lines above as example input, an example of what I would like to have as output would be to only show the srcip, categoryname and url tags in that order. The desired output would look something like this:

10.11.12.13 Uncategorized https://website.net/
10.13.14.15 Education/Reference http://host.com/mini_banner.png

I am looking for a solution that is easily adaptable so on the fly I can tweak which tags are displayed.

  • 1
    The script in my answer should handle spaces in the value part. Please update your post above with an example input line that doesnt work for you. – meuh Nov 30 '17 at 17:50
  • @meuh I ran the sample log lines provided in the original post through your script using the country and ua keys. The returned output was blank. – Andrew S Nov 30 '17 at 18:50
  • It works for me. Make sure you are using gnu awk, or gawk. I dont see where a space might break the match. – meuh Nov 30 '17 at 18:54
  • 1
    I'm using gawk 4.1.3, so that might explain the problem. Looks like your version doesnt support FPAT yet. – meuh Nov 30 '17 at 18:59
  • I've updated my answer with a version without FPAT, though I don't have a pre-4.0 awk to test it on. – meuh Nov 30 '17 at 19:18
2

Your data is highly structured as key="value", so you can write a small shell script using gnu awk which takes as argument a list of key names and just prints those values. Eg, myscript:

#!/bin/bash
awk -v lhs="$*" '
BEGIN{  FPAT = "[a-z-]*=\"[^\"]*\""
        nwant = split(lhs,want)
}
{       for(i=1;i<=NF;i++){
            start = match($i,/([a-z-]*)="([^"]*)"/,a)
            key[a[1]] = a[2]
        }
        for(i=1;i<=nwant;i++){printf "%s ",key[want[i]]; key[want[i]] = ""}
        printf "\n"
}'

which you call as myscript srcip categoryname url. This sets the awk variable lhs to the arguments as a single string, which are split into array want at the beginning. The lines are divided by awk into fields matching the pattern key="value" by using the builtin FPAT variable.

On each line, for each field we split it with match() into 2 captured groups, for the key and for the part in double-quotes. These are put by awk in array a, and we save them in an associative array key indexed by the key string.

Then for each wanted key, we print the value, and clear it for the next line (in case that line does not have this key). Obviously, this assumes all the data has the required structure, and will need changes to handle (") inside the value, or keys with non-alphabetic characters.


Versions of gnu awk (gawk) earlier than 4.0 do not have the FPAT builtin to split the line into fields matching a pattern, so you have to do it yourself:

#!/bin/bash
awk -v lhs="$*" '
BEGIN{ nwant = split(lhs,want) }
{       input = $0
        while(match(input,"[a-z-]*=\"[^\"]*\"")>0){
            field = substr(input,RSTART,RLENGTH)
            input = substr(input,RSTART+RLENGTH)
            start = match(field,/([a-z-]*)="([^"]*)"/,a)
            key[a[1]] = a[2]
        }
        for(i=1;i<=nwant;i++){printf "%s ",key[want[i]]; key[want[i]] = ""}
        printf "\n"
}'

Obviously, you could combine the two match calls into one, but this shows the difference with the original.

  • You can change this line start = match($i,/([a-z-]*)="([^"]*)"/,a) to this split($i, a, "=");. The result will be almost the same, except that outer double quotes will be left. But may be it is good, because of values with spaces, like: ua=Mozilla/5.0 (X11; CrOS x86_64 9765.85.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.123 Safari/537.36. These cases are more readable with double quotes. – MiniMax Nov 30 '17 at 16:40
  • 1
    MiniMax, your split change works great however it doesn't work correctly when URLs contain equal signs as in this case: url="7.freeaddon.com/v1/geo/?uid=2sadfsa568-fgce-0gb2" – Andrew S Nov 30 '17 at 16:56
  • @AndrewS Then you can do this: split($i, a, "=\""). Splitting not by equal sign, but equal + double quotation mark. But then another problem - the single quotation mark leaves in the end of the value. – MiniMax Nov 30 '17 at 17:12
  • @AndrewS One solution is return the missing quotation mark, after the split function remove it: split($i, a, "=\""); key[a[1]] = "\"" a[2]. – MiniMax Nov 30 '17 at 17:21
  • @MiniMax Your solution is definitely an improvement. However, after running several tests, the original script by meuh has a flaw where it doesn't correctly handle values with spaces. In the original script by meuh values with spaces are returned as empty and your modified version returns just the first word with no closing quotation mark. – Andrew S Nov 30 '17 at 17:43
1

Using (POSIX-compliant) sed...

sed 's/.* srcip="\([^"]*\)" .* url="\([^"]*\)" .* categoryname="\([^"]*\)" .*/\1 \3 \2/' logfile

Nothing fancy here, just find the keys and surround the values with parens \(..\) which allows them to be used as back-references. Then we substitute for the string with just the back-refs, space-delimited, ordered per your requirement: \1 \3 \2.

Output:

10.11.12.13 Uncategorized https://website.net/
10.13.14.15 Education/Reference http://host.com/mini_banner.png

If the logs contain strings that don't have all of these keys then you can use:

sed -n 's/.* srcip="\([^"]*\)" .* url="\([^"]*\)" .* categoryname="\([^"]*\)" .*/\1 \3 \2/p' logfile

This will only print lines that match the pattern.

And, of course, if you want to use these in a streaming fashion just remove the filename and do [something sending logs to stdout] | sed ...

  • Your sed solution works well. I had previously tried using sed but the part I was missing was the correct regular expression of [^"] to match the next quotation mark versus skipping to the last quotation mark. Also the -n to only show matches was very helpful. Thanks! – Andrew S Nov 30 '17 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.