2

I have big file that holds something like 40000 rows of domain names. I would like to read that file and use dig (or something else) to look up the IP addresses of the domain names in the DNS, and print them out to another file.

How do I do this?

EDIT: Been testing this with some of the proposed solutions. With this for most the part:

#!/bin/bash
> ips.txt
cat test.txt | while read host; do
    ip=$(getent hosts "$host")
    if [ $? -ne 0 ]; then
        echo "Host $host was not resolved.";
        continue
    fi
    ip=$(echo "$ip" | awk '{ print $1 }')
    echo "Host: $host, IP: $ip" >> ips.txt
done

This produces a file that is empty. Not sure why this is not working.

I tried another solution:

for host in 0.accountkit.com 0.bigclicker.me 0.cdn.ddstatic.com 0.facebook.com 0.fls.doubleclick.net 0.hiveon.net 0.mining.garden
do
    # get ips to host using dig
    ips=($(dig "$host" a +short | grep '^[.0-9]*$'))
    for ip in "${ips[@]}";
    do
        printf 'allow\t\t%s\n' "$ip"
    done
done > allowedip.txt

This will print the ip-addresses but problem is that I need to read the DNS names from the file and not in the script itself.

2
  • Something like for i in $(cat dns_file); do dig $i >> new_dns_file; done ? – mehlj Dec 23 '19 at 13:56
  • 1
    hi, no dig is not a requirement :) – Toube Dec 23 '19 at 13:57
0

Another loop. This one reads a list of hostnames from hosts and writes each hostname and its zero or more IPv4 addresses to ips. I've separated the host from its list of IP addresses with a tab (\t), and each IP address is separated from the next with a space:

#!/bin/bash
while IFS= read -r host
do
    if [[ -n "$host" ]]
    then
        ips=$(dig +short "$host" | grep '^[[:digit:].]*$' | xargs)
        printf "%s\t%s\n" "$host" "$ips"
    fi
done <hosts >ips

Example data:

Source file hosts

bbc.co.uk
google.co.uk

Results file ips

bbc.co.uk       151.101.192.81 151.101.128.81 151.101.64.81 151.101.0.81
google.co.uk    216.58.213.3
1
1

It's good that you say that dig is not a requirement, because it is not the best tool for the job.

Tools like the host command (available in 3 implementations for many operating systems: the one from ISC's BIND, the one from Knot DNS, and the one from djbwares) print the information in human-readable form, and this has to be tediously post-processed to remove chaff if one wants a simple machine-readable list.

ISC's dig and Knot DNS kdig have a +short option, which improves upon this (and the ordinary outputs of dig/kdig), but are limited to being invoked on one domain name at a time, necessitating a loop in shell script and at least 40,000 processes to do the job. (It's about 160,000 processes in one of the answers here.)

Another tool for this is Daniel J. Bernstein's dnsip, part of his djbdns toolset. What you want to do is a one-liner via xargs, because the tool can take multiple domain name arguments:

% cat domain_names.list
unix.stackexchange.com
freebsd.org
cr.yp.to
%
% xargs dnsip < domain_names.list
151.101.65.69 151.101.193.69 151.101.1.69 151.101.129.69
96.47.72.84
131.193.32.109 131.193.32.108
%

This reduces the process count somewhat. (A quick back-of-the-envelope test shows that the reduction is by 3 orders of magnitude. My test list of 40,000 domain names resulted in just 9 dnsip processes.)

There's a dnsipq tool for when one wants to use non-fully-qualified domain names:

% cat domain_names.list
unix
freebsd
crypto
%
% xargs dnsipq < domain_names.list
unix.stackexchange.com 151.101.1.69 151.101.193.69 151.101.65.69 151.101.129.69
freebsd.stackexchange.com 151.101.65.69 151.101.193.69 151.101.1.69 151.101.129.69
crypto.stackexchange.com 151.101.1.69 151.101.129.69 151.101.193.69 151.101.65.69
%

Further reading

3
  • thanks, I haven't tried this one as I need to get the IP:s with line breaks. And I'm not sure how to use this? – Toube Dec 24 '19 at 11:44
  • Turning the spaces into linefeeds is a fairly trivial exercise in the use of tr. – JdeBP Dec 24 '19 at 15:19
  • Thanks, got it working with roaimas solution – Toube Dec 24 '19 at 20:15
0

Either in a single command line or a tiny bash script:

while IFS= read -r Domainname; do [ -n "$Domainname" ] && echo "$Domainname: $(dig +short "$Domainname" | tr '\n' ' ')"; done < domainlist.csv

This read the file domainlist.csv containing the domain names into the while loop. The body just outputs all non-empty lines.

3
  • Thanks all, I will soon see the result and let you know if it worked out. – Toube Dec 23 '19 at 15:05
  • also works but output is also an empty file. – Toube Dec 24 '19 at 11:43
  • So looks like this is working: I removed the spaces between the outputs: while IFS= read -r Domainname; do [ -n "$Domainname" ] && echo "$(dig +short "$Domainname")"; done < test.txt > ips.txt. Though this will print the not found values as dns names.. so if I just could get the ip:s that are found to be printed this would work. – Toube Dec 24 '19 at 13:42
0

This would take each line in file hosts.txt, do a DNS lookup on the host and write the resultant IP address (alongside the hostname) to ips.txt. Keep in mind that it doesn't do parallel lookups to reduce time nor is it geared to handle invalid inputs. And it doesn't handle multiple IP addresses.

As for errors, if a hostname is not resolved, it will print out an error to the screen. Nothing will be written to ips.txt for that host.

#!/bin/bash
> ips.txt
cat hosts.txt | while read host; do
    ip=$(getent hosts "$host")
    if [ $? -ne 0 ]; then
        echo "Host $host was not resolved.";
        continue
    fi
    ip=$(echo "$ip" | awk '{ print $1 }')
    echo "Host: $host, IP: $ip" >> ips.txt
done
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.