Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to use a bash script to process a webserver log file and replace any IP's it finds with their corresponding DNS hostnames.

An example entry of a single line from the log file is:

<12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=192.168.1.6 DST=192.168.1.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51 

(I have changed all private details in the above line for example purposes).

So above, the two fields SRC=192.168.1.6 and DST=192.168.1.1 contain IP addresses, that I need to convert into DNS hostnames (I understand they are just internal addresses, this is just as an example).

This is what I have come up with so far for my script:

#!/bin/bash

logFile=$1

while read line
do
    for word in $line
    do

            # if word is ip address change to hostname
            if [[ $word =~ 'DST='^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
            then
                    # check if ip address is correct
                    ip=($word) | cut -d'=' -f 2
                    echo -n `nslookup $word | grep Name | cut -d' ' -f 8`
                    echo -n " "
            # else print word
            else
                    echo -n $word
                    echo -n " "
            fi
    done
    # new line
    echo
done < "$logFile"

The part that is throwing me is interpreting the DST= and SRC= fields as an IP address, I'm not really sure of the syntax to strip this off prior to DNS processing, then adding it back on following DNS processing, or if there is a better way?

I did search the forums in advance and found the following article: resolve all ip addresses in command output using standard command line tools

However it didn't seem to work, potentially given the format of my log files.

share|improve this question

2 Answers 2

@Dave, Check the script below:

Sample input file

[mihai@image-host-1 tmp]$ cat demo.log
    <12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=166.78.125.161 DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
    <12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=166.78.125.162 DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
    <12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=166.78.125.163 DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
    <12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=166.78.125.164 DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
    <12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=hostnamesrc DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
    <12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=166.78.125.164 DST=hostnamedst LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
    <12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=hostnamesrc DST=hostnamedst LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
  • Contains valid (DNS resolvable IP)
  • Contains invalid IP (*.61. *.63)
  • Contains hostname as SRC

Sample output

[mihai@image-host-1 tmp]$ ./demo.sh demo.log
<12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=NODNS-166.78.125.161 DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
<12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=bangimage.com. DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
<12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=NODNS-166.78.125.163 DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
<12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=NODNS-166.78.125.164 DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
<12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=hostnamesrc DST=173.194.46.38 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
<12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=NODNS-166.78.125.164 DST=hostnamedst LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51
<12>1 2013-11-04T15:04:05+00:00 networkname kernel - - - kernel: [161030.740000] ACCEPT IN=br0 OUT= MAC=00:11:22:33:44:11:00:11:11:11:11:11:11:11 SRC=hostnamesrc DST=hostnamedst LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=30324 DF PROTO=UDP SPT=43729 DPT=53 LEN=51

Actual script

[mihai@image-host-1 tmp]$ cat demo.sh
#!/bin/bash

logFile=$1

while read logLine
do
        # For each log line, find the SRC
        # If needed, this can be extended to DSC as well
        # ----------------------------------------------
        logSRC=`echo $logLine | awk '{print $14}' | awk -F "=" '{print $2}'`
        # echo "SRC = ${logSRC}"

        # Test if SRC is an IP or not
        # ---------------------------
        if [[ ${logSRC} =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
        then
                # echo "${logSRC} is IP"

                # Convert IP into hostname via reverse DNS lookup
                # -----------------------------------------------
                logSRCHOST=`host ${logSRC} | awk '{print $NF}'`

                if [[ ${logSRCHOST} =~ 'NXDOMAIN' ]];
                then
                        logSRCHOST="NODNS-${logSRC}"
                fi
        else
                logSRCHOST=${logSRC}

        fi

        # echo "FINAL SRC = ${logSRCHOST}"

        echo $logLine | sed -e "s/SRC.*DST/SRC=${logSRCHOST} DST/g"

done < "$logFile"
share|improve this answer
    
MTIhai, this is brilliant, thanks very much for your help. I've understood how you've implemented everything so that I can include the DST as well, I'm just a little stuck with the very last line though: echo $logLine | sed -e "s/SRC.*DST/SRC=${logSRCHOST} DST/g" The field after DST is LEN, however later on in the line there is another LEN, which has caused problems whenever I've tried to insert the text back in. The closest I've been able to get is: echo $logLine | sed -e "s/DST=${logDST}/DST=${logDSTHOST}/" However it doesn't seem to reliably insert on all lines. Any thoughts? –  Dave Nov 8 '13 at 18:03
    
I will need to see a sample of your log file, and the output of the script (mine or the version you modified). Can you add them in the question body ? –  MTIhai Nov 13 '13 at 10:18
    
Also, can you please let us know what exactly is LEN suppose to "count" ? The size of the IP/Host from SRC or DST ? . If so, after modifying the SRC/DST, you can assign the LEN columns the values of $#{SRC VARIABLE} or $#{DST VARIABLE}. –  MTIhai Nov 13 '13 at 10:21

logresolve

If these are Apache logs you can use the tool logresolve which is included with Apache to do this work for you.

alternative shell script

I also came across this SO Q&A titled: Replace IPs with Hostnames in a log. The accepted answer to this question included the following shell script for converting IPs to DNS hostnames.

#!/bin/bash

logFile=$1

while read line
do
  for word in $line
  do
    # if word is ip address change to hostname
    if [[ $word =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
    then
      # check if ip address is correct
      OIFS=$IFS
      IFS="."
      ip=($word)
      IFS=$OIFS
      if [[ ${ip[0]} -le 255 && ${ip[1]} -le 255 && ${ip[2]} -le 255 && ${ip[3]} -le 255 ]]
      then
        echo -n `host $word | cut -d' ' -f 5`
        echo -n " "
      else
        echo -n "$word"
        echo -n " "
      fi
    # else print word
    else
        echo -n $word
        echo -n " "
    fi
  done
  # new line
  echo
done < "$logFile"

Save the above script to the filename ip_to_hostname.sh and then run it like so:

$ ./ip_to_hostname.sh your_logfile > resolved_ip
share|improve this answer
    
Thanks for answering. Unfortunately these are not Apache logs, they are DD-WRT logs. From the looks of the script you provided I think that's one of the original scripts I've used to butcher together my script, when I use the original script it processes the log file but the output file is exactly the same as the input file. I think it may be because the log format is written as SRC=1.2.3.4, so it reads the whole string as a word, instead of just the IP address. –  Dave Nov 4 '13 at 19:32
    
@Dave - OK, if I get a chance later tonight I'll take a look and help you out. Unless someone else jumps in here first. –  slm Nov 4 '13 at 19:51
    
No problem friend. I'll keep at it myself and will post the corrections if I figure it out. –  Dave Nov 4 '13 at 20:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.