I have a script that I built .. And it works fantastically -- BUT it's going to take an estimated 4 days to run! I was wondering if there's a more efficient way of doing this.

Here's what the script does:

  1. It gets all files from imageserver and loads them into imageserver.txt
  2. It formats the file paths for grepping
  3. Loops through imageserver.txt and greps /var/www/html per line
  4. Writes to 2 formatted (exist and no) files for later usage
  5. Write to log file for tail to track script progress

What I have is 2 files.

  1. imageserver.txt (about 250,000 lines)

  2. search.sh

    echo "\n\n Started ...\n\n"
    # Clear Runtime Files
    > doesExist.txt
    > nonExists.txt
    > imgSearch.log
    echo "\n\n Building Image List ...\n\n"
    #write contents of imageserver to imageserver.txt
    find /var/www/imageserver/ -type f > imageserver.txt
    # Remove /var/www
    find ./imageserver.txt -type f -readable -writable -exec sed -i "s/\/var\/www\///g" {} \;
    echo "\n\n Finished Building Start Searching ...\n\n"
    linecount=$(wc -l < ./imageserver.txt)
    while IFS= read -r var
    echo "$linecount\n\n"
    echo "\n ... Searching $var\n "
    results=$(grep -rl "$var" /var/www/html)
    if [ $? -eq 0 ]; then
        echo "Image exists ...\n"
        echo "$var|||$results^^^" >> doesExist.txt
        echo "$linecount | YES | $var " >> imgSearch.log
        echo "Image does not exist ... \n"
        echo $var >> nonExists.txt
        echo "$linecount | NO | $var " >> imgSearch.log
    done < ./imageserver.txt
    echo "\n\n -- FINISHED -- \n\n"

Basically I am checking to see if the images are used withing ANY of the html within the /var/www/html directory.

With that said .. Each iteration of grep takes about .5 - 1 second. At my calculations that's 3 - 4 days.. While I SUPPOSE that's exceptable .. Is there a better (more efficient) way of accomplishing this?

1 Answer 1


The performance of the script is not your problem.

You are grepping each find under /var/www/html 250,000 times!

You'll want to replace that while loop with:

grep -rl -F -f ./imageserver.txt /var/www/html > grep_output

And then you'll parse that output file to get your statistics. Which will be tricky, but it won't take 4 days.

or, maybe as simple as

grep -Ff ./imageserver.txt -o grep_output | sort -u

to get the list of images used. You could use comm to compare this against the imageserver.txt to find the images not used.

  • 1
    I thought about this, and this would probably be helped by batching the search. Using grep -f on huge amounts of patterns ofter makes the process run into memory issues.
    – Kusalananda
    Mar 22, 2019 at 20:46
  • Thanks @glenn -- I'll play around with this over the weekend .. It's a good mention also about the -f flag and memory .. I'll be sure to watch top and make sure all is well.
    – Zak
    Mar 22, 2019 at 23:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .