I have to process a file with 1 Million domains lines call:


And now my script that is verifying the lines contained in 1m.txt:

cat out.txt > advance.txt 2> /dev/null
cat 1m.txt | xargs -I {} -P 100 sh -c "if ! grep --quiet {} advance.txt; then if host {} >/dev/null; then echo OK {}; else echo DIE {}; fi; fi" >> out.txt

What this script does is that if it is interrupted (ctrl + c) and restarted, it starts from the last line processed. If there are 1000 lines, the restart is fast when interrupting the process on the line 200. But 1 million lines and interrupting the process on the line 500k takes hours

Is there a way to make it more efficient?

  • 1
    Yes. Not write it in shellscript. Use the tool appropriate for the job. May 14, 2021 at 1:34
  • 2
    You seem to be grepping advance.txt 1 million times, and when you find that a domain was not yet processed, you call host. You could try grep -v -f 1m.txt advance.txt, replacing one million small greps with a single huge grep. The resulting list is the domains that still need processing (please double-check my logic). May 14, 2021 at 1:38
  • @berndbausch thanks I appreciate your contribution
    – acgbox
    May 14, 2021 at 13:05

1 Answer 1


So your current logic is "For each line in 1m.txt, see if it is already in advance.txt. If not then process it and add it to out.txt. When the job is started update advance.txt with all the lines in out.txt'.

The problem with this is that as more lines are added to advance.txt the more lines each line have to be compared. As a worst case if every line had been processed then each of the million lines in 1m.txt will need to be checked to see if it is in advance.txt. On average you need to compare half the lines in advance.txt, so this will require 1,000,000*500,000 or 500,000,000,000 (500 billion) compares.

If you were not processing things in parallel a straight forward way to handle this would be to find the last line in out.txt, and skip all the lines in 1m.txt up to that point. e.g.

# Pipe the output of the if/then/else/fi construct to xargs.
# use the if/then/else/fi to select the input.
# Use '-s' to see if the file exists and has non zero size.
 if [ -s out.txt ] ; then
    # we have some existing data
    # Get the host from the last line
    # delete anything that is not the last line
    # remove the DIE/OK. quote anything not alphabetic with a backslash.
   lasthost="$(sed '$!d;s/^\(DIE\|OK\) //;s/[^0-9a-zA-Z]/\\&/g' out.txt)"
   # get the lines from 1m.txt from after the matched host
   # uses GNU sed extension to start at line "0"
   sed "0,/^$lasthost\$/d" 1m.txt
   # no existing data, so just copy the 1m.txt using cat
   cat 1m.txt
 fi | xargs -I {} sh -c "if host {} >/dev/null; then echo OK {}; else echo DIE {}; fi" >> out.txt

However you are processing things in parallel. Since host can take very variable amounts of time to return a value, the input can be significantly rearranged. A faster way to see if the host has been already seen is needed. The standard way is to use a hash table of some kind. One way is to use awk.

 if [ -s out.txt ] ; then
    # we have some existing data. Process the two files given
    # for the first file set the entries of the seen array to 1
    # for the second file print out the hosts which have not been seen. 
    awk 'FNR==NR {seen[$2]=1;next} seen[$1]!=1' out.txt 1m.txt
   cat 1m.txt
 fi | xargs -I {} -P 100 sh -c "if host {} >/dev/null; then echo OK {}; else echo DIE {}; fi" >> out.txt
  • magnificent, thanks
    – acgbox
    May 14, 2021 at 12:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .