1

I have a script that I built .. And it works fantastically -- BUT it's going to take an estimated 4 days to run! I was wondering if there's a more efficient way of doing this.

Here's what the script does:

  1. It gets all files from imageserver and loads them into imageserver.txt
  2. It formats the file paths for grepping
  3. Loops through imageserver.txt and greps /var/www/html per line
  4. Writes to 2 formatted (exist and no) files for later usage
  5. Write to log file for tail to track script progress


What I have is 2 files.

  1. imageserver.txt (about 250,000 lines)

    imageserver/icons/socialmedia/sqcolor_tumblr.png
    imageserver/icons/socialmedia/sqcolor_gaf.png
    imageserver/icons/socialmedia/sqcolor_yelp.png
    imageserver/icons/socialmedia/sqcolor_linkedin.png
    imageserver/icons/socialmedia/sqcolor_twitter.png
    imageserver/icons/socialmedia/sqcolor_angies.png
    imageserver/icons/socialmedia/sqcolor_houzz.png
    
  2. search.sh

    #!/bin/bash
    
    echo "\n\n Started ...\n\n"
    
    # Clear Runtime Files
    > doesExist.txt
    > nonExists.txt
    > imgSearch.log
    
    echo "\n\n Building Image List ...\n\n"
    
    #write contents of imageserver to imageserver.txt
    find /var/www/imageserver/ -type f > imageserver.txt
    
    # Remove /var/www
    find ./imageserver.txt -type f -readable -writable -exec sed -i "s/\/var\/www\///g" {} \;
    echo "\n\n Finished Building Start Searching ...\n\n"
    
    linecount=$(wc -l < ./imageserver.txt)
    
    while IFS= read -r var
    do
    echo "$linecount\n\n"
    echo "\n ... Searching $var\n "
    
    results=$(grep -rl "$var" /var/www/html)
    if [ $? -eq 0 ]; then
        echo "Image exists ...\n"
        echo "$var|||$results^^^" >> doesExist.txt
        echo "$linecount | YES | $var " >> imgSearch.log
    else
        echo "Image does not exist ... \n"
        echo $var >> nonExists.txt
        echo "$linecount | NO | $var " >> imgSearch.log
    fi
    
    linecount=$((linecount-1))
    done < ./imageserver.txt
    
    echo "\n\n -- FINISHED -- \n\n"
    

Basically I am checking to see if the images are used withing ANY of the html within the /var/www/html directory.

With that said .. Each iteration of grep takes about .5 - 1 second. At my calculations that's 3 - 4 days.. While I SUPPOSE that's exceptable .. Is there a better (more efficient) way of accomplishing this?

3

The performance of the script is not your problem.

You are grepping each find under /var/www/html 250,000 times!

You'll want to replace that while loop with:

grep -rl -F -f ./imageserver.txt /var/www/html > grep_output

And then you'll parse that output file to get your statistics. Which will be tricky, but it won't take 4 days.

or, maybe as simple as

grep -Ff ./imageserver.txt -o grep_output | sort -u

to get the list of images used. You could use comm to compare this against the imageserver.txt to find the images not used.

  • 1
    I thought about this, and this would probably be helped by batching the search. Using grep -f on huge amounts of patterns ofter makes the process run into memory issues. – Kusalananda Mar 22 at 20:46
  • Thanks @glenn -- I'll play around with this over the weekend .. It's a good mention also about the -f flag and memory .. I'll be sure to watch top and make sure all is well. – Zak Mar 22 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.