4

I would like to verify if a URL exists without downloading. I am using below with curl:

if [[ $(curl ftp://ftp.somewhere.com/bigfile.gz) ]] 2>/dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

or using wget:

if [[ $(wget ftp://ftp.somewhere.com/bigfile.gz) -O-]] 2>/dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

This works great if the URL doesn't exist. If it exists, it downloads the file. In my case, the files are really big and I do not want it to download. I just want to know if that URL exists.

1
  • If you are trying to save disk space, try redirecting output of the command to /dev/null.
    – Lewis M
    Oct 11 '18 at 11:48
9

You're close. The correct way to approach this is using the HEAD method.

With cURL:

if curl --head --silent --fail ftp://ftp.somewhere.com/bigfile.gz 2> /dev/null;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi

or using wget:

if wget -q --method=HEAD ftp://ftp.somewhere.com/bigfile.gz;
 then
  echo "This page exists."
 else
  echo "This page does not exist."
fi
4
  • what version of wget did you use for this? I get the error: wget: unrecognized option '--method=HEAD' Jan 6 '20 at 15:27
  • --method=HEAD was added in Wget 1.15 released over 5 years ago.
    – darnir
    Jan 7 '20 at 10:06
  • thanks looks like we have version 1.14 on our CentOS 7 server (Build Date: 2016-06-03), that would explain why we dont have this feature Jan 7 '20 at 18:07
  • 1
    You could use the --spider option in that case.
    – darnir
    Jan 8 '20 at 8:10
2

Try curl --head (HTTP FTP FILE) Fetch the headers only!

status=$(curl --head --silent ftp://ftp.somewhere.com/bigfile.gz | head -n 1)

if echo "$status" | grep -q 404
  echo "file does not exist"
else
  echo "file exists"
fi
1
if curl --output /dev/null --silent --head --fail "ftp://ftp.somewhere.com/bigfile.gz"
then
  echo "This page exists."
 else
  echo "This page does not exist."
fi
1

I think, it is better to use the --spider argument with the wget tool. It also works on lightweight versions of the wget tool (in BusyBox).

Example:

URL="http://www.google.com"

if wget --spider "${URL}" 2>/dev/null; then
    echo "OK !"
else
    echo "ERROR ! Online URL ${URL} not found !"
fi
-1
echo $(curl --head --silent <url> | head -n 1)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.