Progress indicator when using a for loop to parse txt file

Question

I'm trying to create a really simple script that tests dns servers and returns the results into a csv file, and it works perfectly, but takes a long time with a large source file of domains.

Is there some way to use pv or awk to create a progress indicator?

#!/bin/bash
# File name of domain list: One FQDN per line in file.
domain_list='domains.txt'
#
# IP address of the nameserver used for lookups:
ns1_ip='1.1.1.1' # Cloudflare
ns2_ip='9.9.9.9' # Quad9
#
# Seconds to wait between lookups:
loop_wait='1'
#
echo "Domain name, $ns1_ip,$ns2_ip" > dns-test-results.csv;
for domain in `cat $domain_list`
do
    ip1=`dig @$ns1_ip +short $domain |tail -n1`; 
    ip2=`dig @$ns2_ip +short $domain |tail -n1`;
    echo -en "$domain,$ip1,$ip2\n" >> dns-test-results.csv;
# 
done;

Answer 1

You could store the items in an array first to count them, and then just print the number of processed items and the total on each loop iteration.

E.g. something like this:

items=( $(cat items.txt) )
i=0; for x in "${items[@]}"; do
    printf "\r%d/%d" "$(( i += 1 ))" "${#items[@]}";
    sleep 1  # do some actual work here
done
echo

Though note that relying on word-splitting of $(cat file...) is somewhat problematic, and you might be better off just using readarray to read each input file into an array element more safely:

readarray -t items < items.txt

Answer 2

Similarly to what ilkkachu suggested, you can just read with a while loop:

#!/bin/bash
# File name of domain list: One FQDN per line in file.
domain_list='domains.txt'
number=$( wc -l < "$domain_list")
#
# IP address of the nameserver used for lookups:
ns1_ip='1.1.1.1' # Cloudflare
ns2_ip='9.9.9.9' # Quad9
#
# Seconds to wait between lookups:
loop_wait='1'
#
echo "Domain name, $ns1_ip,$ns2_ip" > dns-test-results.csv;
count=0
while read -r domain
do
  (( count++ ))
  printf '\rProcessing domain %d of %d' "$count" "$number"
  ip1=$(dig @$ns1_ip +short $domain |tail -n1); 
  ip2=$(dig @$ns2_ip +short $domain |tail -n1);
  printf "%s,%s,%s\n" "$domain" "$ip1" "$ip2" >> dns-test-results.csv;
  sleep "$loop_wait"
# 
done < "$domain_list"
echo

The code above will print Processing domain x of y where x is the current number being looked at and y the total number of lines in $domains_list.

Answer 3

You could add an printf . to your script each time a record has been checked. And pipe through pv like (assuming domains.txt is line separated per record):

script.sh | pv -p -s "$(wc -l < domains.txt)" > /dev/null

But you can make even nicer.

Instead of echo -en "$domain,$ip1,$ip2\n" >> dns-test-results.csv;, you can just have echo "$domain,$ip1,$ip2". Then use the pv line mode and invoke in a more "separation of concerns" manner:

script.sh | pv -p -l -s "$(wc -l < domains.txt)" > dns-test-results.csv

Answer 4

You can cause the shell to print progress info at regular intervals using traps.

#!/bin/bash

# child sends SIGUSR1 to the parent every $PERIOD seconds.

MAINPID=$BASHPID
PERIOD=3

pinger()  {
    (
    trap exit ERR
    while true
    do
        sleep $PERIOD
        kill -USR1 $MAINPID
    done
    ) &
}

# ====

# when main process receives SIGUSR1, print some progress info

trap progress SIGUSR1

progress() {
    printf "Progress: %d spins are spun\r" $COUNT
    return
}

# ====

# Start the pinger
pinger

# actual work goes here

for((COUNT=1; COUNT<300; COUNT++))
do
    sleep 0.100
done

exit