I have a script that I built .. And it works fantastically -- BUT it's going to take an estimated 4 days to run! I was wondering if there's a more efficient way of doing this.

Here's what the script does:

  1. It gets all files from imageserver and loads them into imageserver.txt
  2. It formats the file paths for grepping
  3. Loops through imageserver.txt and greps /var/www/html per line
  4. Writes to 2 formatted (exist and no) files for later usage
  5. Write to log file for tail to track script progress

What I have is 2 files.

  1. imageserver.txt (about 250,000 lines)

  2. search.sh

    echo "\n\n Started ...\n\n"
    # Clear Runtime Files
    > doesExist.txt
    > nonExists.txt
    > imgSearch.log
    echo "\n\n Building Image List ...\n\n"
    #write contents of imageserver to imageserver.txt
    find /var/www/imageserver/ -type f > imageserver.txt
    # Remove /var/www
    find ./imageserver.txt -type f -readable -writable -exec sed -i "s/\/var\/www\///g" {} \;
    echo "\n\n Finished Building Start Searching ...\n\n"
    linecount=$(wc -l < ./imageserver.txt)
    while IFS= read -r var
    echo "$linecount\n\n"
    echo "\n ... Searching $var\n "
    results=$(grep -rl "$var" /var/www/html)
    if [ $? -eq 0 ]; then
        echo "Image exists ...\n"
        echo "$var|||$results^^^" >> doesExist.txt
        echo "$linecount | YES | $var " >> imgSearch.log
        echo "Image does not exist ... \n"
        echo $var >> nonExists.txt
        echo "$linecount | NO | $var " >> imgSearch.log
    done < ./imageserver.txt
    echo "\n\n -- FINISHED -- \n\n"

Basically I am checking to see if the images are used withing ANY of the html within the /var/www/html directory.

With that said .. Each iteration of grep takes about .5 - 1 second. At my calculations that's 3 - 4 days.. While I SUPPOSE that's exceptable .. Is there a better (more efficient) way of accomplishing this?


The performance of the script is not your problem.

You are grepping each find under /var/www/html 250,000 times!

You'll want to replace that while loop with:

grep -rl -F -f ./imageserver.txt /var/www/html > grep_output

And then you'll parse that output file to get your statistics. Which will be tricky, but it won't take 4 days.

or, maybe as simple as

grep -Ff ./imageserver.txt -o grep_output | sort -u

to get the list of images used. You could use comm to compare this against the imageserver.txt to find the images not used.

  • 1
    I thought about this, and this would probably be helped by batching the search. Using grep -f on huge amounts of patterns ofter makes the process run into memory issues. – Kusalananda Mar 22 at 20:46
  • Thanks @glenn -- I'll play around with this over the weekend .. It's a good mention also about the -f flag and memory .. I'll be sure to watch top and make sure all is well. – Zak Mar 22 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.