11

I would like to verify if a URL exists without downloading. I am using below with curl:

if [[ $(curl ftp://ftp.somewhere.com/bigfile.gz) ]] 2>/dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

or using wget:

if [[ $(wget ftp://ftp.somewhere.com/bigfile.gz) -O-]] 2>/dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

This works great if the URL doesn't exist. If it exists, it downloads the file. In my case, the files are really big and I do not want it to download. I just want to know if that URL exists.

2

5 Answers 5

18

You're close. The correct way to approach this is using the HEAD method.

With cURL:

if curl --head --silent --fail ftp://ftp.somewhere.com/bigfile.gz 2> /dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

or using wget:

if wget -q --method=HEAD ftp://ftp.somewhere.com/bigfile.gz;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi
4
  • what version of wget did you use for this? I get the error: wget: unrecognized option '--method=HEAD' Jan 6, 2020 at 15:27
  • --method=HEAD was added in Wget 1.15 released over 5 years ago.
    – darnir
    Jan 7, 2020 at 10:06
  • thanks looks like we have version 1.14 on our CentOS 7 server (Build Date: 2016-06-03), that would explain why we dont have this feature Jan 7, 2020 at 18:07
  • 2
    You could use the --spider option in that case.
    – darnir
    Jan 8, 2020 at 8:10
5

I think, it is better to use the --spider argument with the wget tool. It also works on lightweight versions of the wget tool (in BusyBox).

Example:

URL="http://www.google.com"

if wget --spider "${URL}" 2>/dev/null; then
    echo "OK !"
else
    echo "ERROR ! Online URL ${URL} not found !"
fi
2

Try curl --head (HTTP FTP FILE) Fetch the headers only!

status=$(curl --head --silent ftp://ftp.somewhere.com/bigfile.gz | head -n 1)

if echo "$status" | grep -q 404
  echo "file does not exist"
else
  echo "file exists"
fi
1
if curl --output /dev/null --silent --head --fail "ftp://ftp.somewhere.com/bigfile.gz"
then
  echo "This page exists."
 else
  echo "This page does not exist."
fi
-1
echo $(curl --head --silent <url> | head -n 1)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .