1

I have a text file containing a list of URLs like this:

http://somesite.com/some-article/
https://anothersite.fr/somepage.php?page=something

What I'd like to do is check which URLs return HTTP code 200 (OK) and put them in a list in another file.

How could I accomplish this using terminal? Curl?

1

Try doing this using :

while read url ; do
    ret=$(curl -I -s "$url" -o /dev/null -w "%{http_code}\n")
    ((ret==200)) && echo "$url" >> new_list_URL
done < list_URL

or POSIX mode :

while read url ; do
    ret=$(curl -I -s "$url" -o /dev/null -w "%{http_code}\n")
    [ $ret -eq 200 ] && echo "$url" >> new_list_URL
done < list_URL

Check man curl

  • Thank you sputnick, but I can't seem to make it work. For some reason it doesn't output to new_list_URL. What could be the problem? – Eddie M. Patrick Dec 28 '14 at 11:46
0

May not be the best way, but you can use this easy-to-understand shell script too:

while read i
do
wget --spider $i > /dev/null 2>1
if [ $? == 0 ]
then
echo $i >> validlist.txt
fi
done

Run this shell script as ./wget_check.sh < urllist.txt , where wget_check.sh is the script name and urllist.txt is the text file holding the URLs.

This script basically runs a wget --spider against all each of the URLs in the list and will add the URL to another file 'validlist.txt' if the URL returns a valid 200 OK response.

  • The --spider option will 'crawl' through the URLs and will not download any files.

  • There will be no output produced since the output will be redirected to /dev/null.

  • Each wget will return a non-zero return value if it does not get a 200 OK response

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.