May not be the best way, but you can use this easy-to-understand shell script too:
while read i
wget --spider $i > /dev/null 2>1
if [ $? == 0 ]
echo $i >> validlist.txt
Run this shell script as
./wget_check.sh < urllist.txt , where
wget_check.sh is the script name and urllist.txt is the text file holding the URLs.
This script basically runs a
wget --spider against all each of the URLs in the list and will add the URL to another file 'validlist.txt' if the URL returns a valid 200 OK response.
--spider option will 'crawl' through the URLs and will not download any files.
There will be no output produced since the output will be redirected to /dev/null.
wget will return a non-zero return value if it does not get a
200 OK response