2

I have a list of URLs in file urls.txt.

If possible, I want to get all URLs that give a 404 error when I attempt to fetch them with curl, and copy them to a new file.

For example the URLs in my file urls.txt:

mysite.com/page1
mysite.com/page2
mysite.com/page3
mysite.com/page4
mysite.com/page5
...
mysite.com/page100
...
mysite.com/page1000

so I want to try to fetch each one and if the fetching fails with error 404, I want to store the failing URL into a new file.

0

This may not be the best, but try this:

Make a file urlcheck.sh, then give permission to execute. or simply enter this command:

touch urlcheck.sh
chmod +x urlcheck.sh

Paste below script to urlcheck.sh

#!/bin/bash
TIMEOUT=3

if [ ! -f output404.txt ]; then
    touch output404.txt
fi

while IFS= read -r line; do
    OUT_URL=$(curl -I $line 2>&1 -m $TIMEOUT| awk '/HTTP\// {print $2}')
    if [ "$OUT_URL" == "404" ]; then
        echo $line >> output404.txt
        echo "$line written to output404.txt"
    else
        echo "$line     $OUT_URL"
    fi
done < "$1"

and save.

To run the script:

./urlcheck.sh urls.txt

then, check output404.txt generated by the script.

Please note that The url in each line must an url readable by curl such as https://unix.stackexchange.com/.

you can change the timeout second in line TIMEOUT=3.

| improve this answer | |
  • thank sir, but output404.txt not generate after runing script – amine May 9 at 20:14
  • then create it first touch output404.txt. I have modified the script. please check. let me know if it works – Jastria Rahmat May 9 at 20:18
  • the fil output 404.txt generate is still empty – amine May 9 at 20:27
  • sorry, my bad, I forgot to rename 404.txt to output404.txt. fixed now. try it – Jastria Rahmat May 9 at 20:30
  • Thank very much sir, now is working, thank a lot – amine May 9 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.