7

I'm trying to create a really simple script that tests dns servers and returns the results into a csv file, and it works perfectly, but takes a long time with a large source file of domains.

Is there some way to use pv or awk to create a progress indicator?

#!/bin/bash
# File name of domain list: One FQDN per line in file.
domain_list='domains.txt'
#
# IP address of the nameserver used for lookups:
ns1_ip='1.1.1.1' # Cloudflare
ns2_ip='9.9.9.9' # Quad9
#
# Seconds to wait between lookups:
loop_wait='1'
#
echo "Domain name, $ns1_ip,$ns2_ip" > dns-test-results.csv;
for domain in `cat $domain_list`
do
    ip1=`dig @$ns1_ip +short $domain |tail -n1`; 
    ip2=`dig @$ns2_ip +short $domain |tail -n1`;
    echo -en "$domain,$ip1,$ip2\n" >> dns-test-results.csv;
# 
done;


2
  • 2
    Check the length of the input, keep a counter how many iterations you completed, print something like 3/20 in CLI (use \r to overwrite)
    – Panki
    Dec 8, 2023 at 8:09
  • Code for progress indicators and spinners in Bash: stackoverflow.com/q/238073/1412534
    – md2perpe
    Dec 9, 2023 at 22:38

4 Answers 4

9

You could store the items in an array first to count them, and then just print the number of processed items and the total on each loop iteration.

E.g. something like this:

items=( $(cat items.txt) )
i=0; for x in "${items[@]}"; do
    printf "\r%d/%d" "$(( i += 1 ))" "${#items[@]}";
    sleep 1  # do some actual work here
done
echo

Though note that relying on word-splitting of $(cat file...) is somewhat problematic, and you might be better off just using readarray to read each input file into an array element more safely:

readarray -t items < items.txt
3
  • 1
    I wouldn't use readarray unless this is a once-off script with known small input. If your file has millions of lines, this will hog memory unnecessarily, and while computers have lots of RAM these days, small VPSs generally don't. Dec 8, 2023 at 15:57
  • @GuntramBlohm, I don't think readarray is the issue there any more than any other way of reading the file to an array. You have to read the input one way or another anyway to count the lines if you want the progress indicator to show the total number of lines/items. (And in any case, the Q had for x in $(cat file), which also reads into memory first.) Of course you could count the lines with wc -l first, and then read it again to process the data, which would save memory at the cost of twice the IO.
    – ilkkachu
    Dec 8, 2023 at 17:50
  • Unnecessary use of cat: items=( $(<items.txt) ) works just as well, ignoring all other issues.
    – Wastrel
    Dec 9, 2023 at 15:07
6

Similarly to what ilkkachu suggested, you can just read with a while loop:

#!/bin/bash
# File name of domain list: One FQDN per line in file.
domain_list='domains.txt'
number=$( wc -l < "$domain_list")
#
# IP address of the nameserver used for lookups:
ns1_ip='1.1.1.1' # Cloudflare
ns2_ip='9.9.9.9' # Quad9
#
# Seconds to wait between lookups:
loop_wait='1'
#
echo "Domain name, $ns1_ip,$ns2_ip" > dns-test-results.csv;
count=0
while read -r domain
do
  (( count++ ))
  printf '\rProcessing domain %d of %d' "$count" "$number"
  ip1=$(dig @$ns1_ip +short $domain |tail -n1); 
  ip2=$(dig @$ns2_ip +short $domain |tail -n1);
  printf "%s,%s,%s\n" "$domain" "$ip1" "$ip2" >> dns-test-results.csv;
  sleep "$loop_wait"
# 
done < "$domain_list"
echo

The code above will print Processing domain x of y where x is the current number being looked at and y the total number of lines in $domains_list.

7
  • 1
    If you put the \r at the beginning of the printf string you can leave the cursor hovering while the lookup's happening. You can also drop a quick tput el (or el=$(tput el) and use "$el") at the end of the printf string so that shorter domain names don't show detritus left by earlier longer names Dec 8, 2023 at 12:22
  • @ChrisDavies I don't really see why having the cursor at the start is better, what a I missing here? The tput makes sense, but seems like overkill since we're only showing numbers. Granted, if we have thousands, that can be a problem but I would fix that by using something like ` printf 'Processing domain %7s of %7s\r' "$count" "$number"`.
    – terdon
    Dec 8, 2023 at 16:36
  • The \r at the front would leave the cursor at the end of the text. And re tput el I misread domain %d as domain %s Dec 8, 2023 at 16:37
  • 1
    Ah yes, I see. That is indeed neater, thanks, fixed.
    – terdon
    Dec 8, 2023 at 16:39
  • This looks like a nice and easy solution to me. Thank you! Even though I have the `\r' my cursor is still on the far left on the P in Processing.
    – 0zero7
    Dec 8, 2023 at 17:31
2

You could add an printf . to your script each time a record has been checked. And pipe through pv like (assuming domains.txt is line separated per record):

script.sh | pv -p -s "$(wc -l < domains.txt)" > /dev/null

But you can make even nicer.

Instead of echo -en "$domain,$ip1,$ip2\n" >> dns-test-results.csv;, you can just have echo "$domain,$ip1,$ip2". Then use the pv line mode and invoke in a more "separation of concerns" manner:

script.sh | pv -p -l -s "$(wc -l < domains.txt)" > dns-test-results.csv
0
1

You can cause the shell to print progress info at regular intervals using traps.

#!/bin/bash

# child sends SIGUSR1 to the parent every $PERIOD seconds.

MAINPID=$BASHPID
PERIOD=3

pinger()  {
    (
    trap exit ERR
    while true
    do
        sleep $PERIOD
        kill -USR1 $MAINPID
    done
    ) &
}

# ====

# when main process receives SIGUSR1, print some progress info

trap progress SIGUSR1

progress() {
    printf "Progress: %d spins are spun\r" $COUNT
    return
}

# ====

# Start the pinger
pinger

# actual work goes here

for((COUNT=1; COUNT<300; COUNT++))
do
    sleep 0.100
done

exit

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .