Read file and print out ip-address to another file

Question

I have big file that holds something like 40000 rows of domain names. I would like to read that file and use dig (or something else) to look up the IP addresses of the domain names in the DNS, and print them out to another file.

How do I do this?

EDIT: Been testing this with some of the proposed solutions. With this for most the part:

#!/bin/bash
> ips.txt
cat test.txt | while read host; do
    ip=$(getent hosts "$host")
    if [ $? -ne 0 ]; then
        echo "Host $host was not resolved.";
        continue
    fi
    ip=$(echo "$ip" | awk '{ print $1 }')
    echo "Host: $host, IP: $ip" >> ips.txt
done

This produces a file that is empty. Not sure why this is not working.

I tried another solution:

for host in 0.accountkit.com 0.bigclicker.me 0.cdn.ddstatic.com 0.facebook.com 0.fls.doubleclick.net 0.hiveon.net 0.mining.garden
do
    # get ips to host using dig
    ips=($(dig "$host" a +short | grep '^[.0-9]*$'))
    for ip in "${ips[@]}";
    do
        printf 'allow\t\t%s\n' "$ip"
    done
done > allowedip.txt

This will print the ip-addresses but problem is that I need to read the DNS names from the file and not in the script itself.

Answer 1

It's good that you say that dig is not a requirement, because it is not the best tool for the job.

Tools like the host command (available in 3 implementations for many operating systems: the one from ISC's BIND, the one from Knot DNS, and the one from djbwares) print the information in human-readable form, and this has to be tediously post-processed to remove chaff if one wants a simple machine-readable list.

ISC's dig and Knot DNS kdig have a +short option, which improves upon this (and the ordinary outputs of dig/kdig), but are limited to being invoked on one domain name at a time, necessitating a loop in shell script and at least 40,000 processes to do the job. (It's about 160,000 processes in one of the answers here.)

Another tool for this is Daniel J. Bernstein's dnsip, part of his djbdns toolset. What you want to do is a one-liner via xargs, because the tool can take multiple domain name arguments:

% cat domain_names.list
unix.stackexchange.com
freebsd.org
cr.yp.to
%
% xargs dnsip < domain_names.list
151.101.65.69 151.101.193.69 151.101.1.69 151.101.129.69
96.47.72.84
131.193.32.109 131.193.32.108
%

This reduces the process count somewhat. (A quick back-of-the-envelope test shows that the reduction is by 3 orders of magnitude. My test list of 40,000 domain names resulted in just 9 dnsip processes.)

There's a dnsipq tool for when one wants to use non-fully-qualified domain names:

% cat domain_names.list
unix
freebsd
crypto
%
% xargs dnsipq < domain_names.list
unix.stackexchange.com 151.101.1.69 151.101.193.69 151.101.65.69 151.101.129.69
freebsd.stackexchange.com 151.101.65.69 151.101.193.69 151.101.1.69 151.101.129.69
crypto.stackexchange.com 151.101.1.69 151.101.129.69 151.101.193.69 151.101.65.69
%

Further reading

• https://unix.stackexchange.com/a/557555/5132
• Daniel J. Bernstein (1999). dnsip. djbwares.
• Daniel J. Bernstein (1999). dnsipq. djbwares.
• https://unix.stackexchange.com/a/446293/5132

Answer 2

Another loop. This one reads a list of hostnames from hosts and writes each hostname and its zero or more IPv4 addresses to ips. I've separated the host from its list of IP addresses with a tab (\t), and each IP address is separated from the next with a space:

#!/bin/bash
while IFS= read -r host
do
    if [[ -n "$host" ]]
    then
        ips=$(dig +short "$host" | grep '^[[:digit:].]*$' | xargs)
        printf "%s\t%s\n" "$host" "$ips"
    fi
done <hosts >ips

Example data:

Source file hosts

bbc.co.uk
google.co.uk

Results file ips

bbc.co.uk       151.101.192.81 151.101.128.81 151.101.64.81 151.101.0.81
google.co.uk    216.58.213.3

Answer 3

Either in a single command line or a tiny bash script:

while IFS= read -r Domainname; do [ -n "$Domainname" ] && echo "$Domainname: $(dig +short "$Domainname" | tr '\n' ' ')"; done < domainlist.csv

This read the file domainlist.csv containing the domain names into the while loop. The body just outputs all non-empty lines.

Answer 4

This would take each line in file hosts.txt, do a DNS lookup on the host and write the resultant IP address (alongside the hostname) to ips.txt. Keep in mind that it doesn't do parallel lookups to reduce time nor is it geared to handle invalid inputs. And it doesn't handle multiple IP addresses.

As for errors, if a hostname is not resolved, it will print out an error to the screen. Nothing will be written to ips.txt for that host.

#!/bin/bash
> ips.txt
cat hosts.txt | while read host; do
    ip=$(getent hosts "$host")
    if [ $? -ne 0 ]; then
        echo "Host $host was not resolved.";
        continue
    fi
    ip=$(echo "$ip" | awk '{ print $1 }')
    echo "Host: $host, IP: $ip" >> ips.txt
done