I am connecting to mySQL database using a shell script. After connecting to the mySQL database, I execute a query. The query gives me 300,000 URLs as the result.

For each of the 300,000 URLs I need to check if the URLs actually exist and need to update the table that the URL is checked for its existence.

I have planned on using the curl command. I am giving the command as below.

curl -s --head http://myurl/ | head -n 1 | grep "HTTP/1.[01] [23].."

If I just give the command in my shell, I am getting the response (like 301, 200 etc). However, I need it in a variable so that I can use it for some manipulation purposes. For example, like below.

$var = curl -s --head http://myurl/ | head -n 1 | grep "HTTP/1.[01] [23].."
echo $var;
if ($var == "some value")
    do something;
    do some other thing;

marked as duplicate by terdon, slm, jasonwryan, Mikel, Braiam Jan 29 '14 at 23:00

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.


To capture the result of a command in a variable, you can use backticks (``) or, better, $(command). In your case, that would be:

var=$(curl -s --head http://myurl/ | head -n 1 | grep "HTTP/1.[01] [23]..")
echo "$var";
if [ "$var" == "some value" ]; then    
    do something;    
    do some other thing;
  • It is not working. I use the above code in a script and when I tried to echo, I am getting "command not found" error in the first line. – Ramesh Jan 29 '14 at 20:56
  • Alright, I was having spaces in my code. Sorry for the confusion. Thanks again for the answer. – Ramesh Jan 29 '14 at 21:00
  • It is working. I had spaces in my bash script. Thanks again. I cannot accept the answer until 4 minutes. So, after that I will accept the answer. – Ramesh Jan 29 '14 at 21:03
  • @Ramesh no worries :). i had posted my previous comment before seeing your reply. – terdon Jan 29 '14 at 21:06

Extracting all 300,000 URLs in one go isn't optimal. You may find it more useful to extract a small number of URLs, and check them, and then extract different URLs.

Let's say you assign status 0 to unchecked URLs. You want to change this status to 200, 301, 404, 403, 401, and so on and so forth.

Let's say each row has a unique ID, which makes things faster.



# Select some rows, selecting ID and URL

# The URL may contain spaces or single quotes.
# I wouldn't trust double quotes; a  ` grep -v '"'  ` piped between
# the mysql --silent and the while could be advisable in this case.

echo "SELECT id, url FROM mytable WHERE status = 0 LIMIT $NUM;" \
    | mysql --silent mydatabase \
    | while [ -n "$ID"  ]; do
    read ID URL
    if [ -n "$ID" ]; then
            # Extract HTTP result code
        CODE=` curl -s --head "$URL" | head -n 1 \
                   | grep "^HTTP/[01]\.[0-9] [1-9][0-9]*" \
                   | cut -f2 -d" " `

            # If there is a code
        if [ -n "$CODE" ]; then
            if [ "$CODE" == "200" ]; then
                echo "200"
                echo "Not 200"
                    # Prepare update
            echo "UPDATE mytable SET status=$CODE WHERE id=$ID;" >> $RESULTS
                 # We might update this ID to a new status in order not to
                 # extract it again.
                 echo "UPDATE mytable SET status=666 WHERE id=$ID;" >> $RESULTS

# update database
mysql mydatabase < $RESULTS
# Remove temporary file

Not the answer you're looking for? Browse other questions tagged or ask your own question.