resolve all ip addresses in command output using standard command line tools

Question

I have several log files that contain a bunch of ip addresses. I would love to be able to pipe the data through a program that would match and resolve ip addresses.

I.E. cat /var/log/somelogfile | host

which would turn a line like

10:45 accessed by 10.13.13.10

into

10:45 accessed by myhostname.intranet

My thought is there might be a way to do this with a combination of sed and host, but I have no idea how to do so. I know that I could write a simple script that would do it, but I would rather be able to use built in tools if possible. Any suggestions?

Answer 1

Here's a quick and dirty solution to this in Python. It does caching (including negative caching), but no threading and isn't the fastest thing you've seen. If you save it as something like rdns, you can call it like this:

zcat /var/log/some-file.gz | rdns
# ... or ...
rdns /var/log/some-file /var/log/some-other-file # ...

Running it will annotate the IP addresses with their PTR records in-place:

$ echo "74.125.132.147, 64.34.119.12." | rdns
74.125.132.147 (rdns: wb-in-f147.1e100.net), 64.34.119.12 (rdns: stackoverflow.com).

And here's the source:

#!/usr/bin/env python

import sys, re, socket

cache = dict()

def resolve(x):
    key = x.group(0)
    try:
        return "%s (rdns: %s)" % (key, cache[key])
    except KeyError:
        try:
            cache[key] = socket.gethostbyaddr(key)[0]
        except socket.herror:
            cache[key] = '?'
        return "%s (rdns: %s)" % (key, cache[key])

for f in [open(x) for x in sys.argv[1:]] or [sys.stdin]:
    for line in f:
        sys.stdout.write(re.sub("\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", resolve, line))

# End of file.

Please note: this isn't quite what you're after to the letter (using ‘standard tools’). But it probably helps you more than a hack that resolves every IP address every time it's encountered. With a few more lines, you can even make it cache its results persistently, which would help with repeat invocations.

Answer 2

I'd use jdresolve -n -a

Packaged for debian, etc. also available at:

https://github.com/jdrowell/jdresolve

    jdresolve resolves IP addresses to hostnames. Any file format is
    supported, including those where the line does not begin with the IP
    address.

I've been using it for over a decade to resolve apache logs, squid logs, and anything else with lots of IP addresses that need resolving. It works well, reliably, and fast, and it can cache lookups from previous runs.

Answer 3

a bash script you can cat your logfile and pipe into.

#!/bin/bash

while read input; do

    for arg in $( echo $input ); do
            match=$(echo "$arg" | grep -P '([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])' )
            if [ "x${match}" = "x" ]; then
                    printf "%-s" "$arg "
            else
                    dns=$( host $arg | tail -1 | awk '{print $NF}' 2>/dev/null )
                    if [ "${dns}" == "3(NXDOMAIN)" ]; then
                            printf "%-s" "$arg "
                    else
                            if [ "x${dns}" == "x" ]; then
                                    printf "%-s" "$arg "
                            else
                                    printf "%-s" "$dns "
                            fi
                    fi
            fi
    done
done
printf "\n"

output looks like:

tk-air:~ tim$ echo "10:45 accessed by 8.8.8.8" | ./get-dns 
10:45 accessed by FWDR-8.FWDR-8.FWDR-8.FWDR-8. 

tk-air:~ tim$ echo "10:45 accessed by 8.8.8.8 26 times" | ./get-dns 
10:45 accessed by FWDR-8.FWDR-8.FWDR-8.FWDR-8. 26 times 

Answer 4

A quick perl one:

perl -MSocket -pe 's/(\d+\.){3}\d+/"$&\[".gethostbyaddr(inet_aton($&), AF_INET)."]"/ge'

Answer 5

If the log format is consistently displaying the same as you show above, you can do this really dirty with echo 10:45 accessed by 10.13.13.10|awk '{print $4}'|nslookup