88

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden).

I can view the file using the web browser on the same machine.

I try again with my browser's user agent, obtained by http://www.whatsmyuseragent.com. I do this:

wget -U 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' http://...

and

curl -A 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' http://...

but it is still forbidden. What other reasons might there be for the 403, and what ways can I alter the wget and curl commands to overcome them?

(this is not about being able to get the file - I know I can just save it from my browser; it's about understanding why the command-line tools work differently)

update

Thanks to all the excellent answers given to this question. The specific problem I had encountered was that the server was checking the referrer. By adding this to the command-line I could get the file using curl and wget.

The server that checked the referrer bounced through a 302 to another location that performed no checks at all, so a curl or wget of that site worked cleanly.

If anyone is interested, this came about because I was reading this page to learn about embedded CSS and was trying to look at the site's css for an example. The actual URL I was getting trouble with was this and the curl I ended up with is

curl -L -H 'Referer: http://css-tricks.com/forums/topic/font-face-in-base64-is-cross-browser-compatible/' http://cloud.typography.com/610186/691184/css/fonts.css

and the wget is

 wget --referer='http://css-tricks.com/forums/topic/font-face-in-base64-is-cross-browser-compatible/' http://cloud.typography.com/610186/691184/css/fonts.css

Very interesting.

Improve this question
4
  • 10
    Pages that check referer are really annoying. The header is supposed to be optional and used for gathering statistics.
    – zaadeh
    Jun 29, 2014 at 10:59
  • The easiest thing I've found is to convert it to a zip file and use it that way.
    – piniyini
    Mar 20, 2015 at 0:00
  • ERROR 403 may also occur when checking timestamps with wget -N. See: stackoverflow.com/a/62176110/658497
    – Noam Manos
    Jun 3, 2020 at 15:10
  • FWIW I had a similar error message when downloading a URL that had access tokens embedded in it, and it was fixed when I enclosed the full URL with quotation marks
    – mostafa.elhoushi
    Sep 3, 2022 at 16:23

5 Answers 5

Reset to default
54

A HTTP request may contain more headers that are not set by curl or wget. For example:

  • Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites. Given a cookie key=val, you can set it with the -b key=val (or --cookie key=val) option for curl.
  • Referer (sic): when clicking a link on a web page, most browsers tend to send the current page as referrer. It should not be relied on, but even eBay failed to reset a password when this header was absent. So yes, it may happen. The curl option for this is -e URL and --referer URL.
  • Authorization: this is becoming less popular now due to the uncontrollable UI of the username/password dialog, but it is still possible. It can be set in curl with the -u user:password (or --user user:password) option.
  • User-Agent: some requests will yield different responses depending on the User Agent. This can be used in a good way (providing the real download rather than a list of mirrors) or in a bad way (reject user agents which do not start with Mozilla, or contain Wget or curl).

You can normally use the Developer tools of your browser (Firefox and Chrome support this) to read the headers sent by your browser. If the connection is not encrypted (that is, not using HTTPS), then you can also use a packet sniffer such as Wireshark for this purpose.

Besides these headers, websites may also trigger some actions behind the scenes that change state. For example, when opening a page, it is possible that a request is performed on the background to prepare the download link. Or a redirect happens on the page. These actions typically make use of Javascript, but there may also be a hidden frame to facilitate these actions.

If you are looking for a method to easily fetch files from a download site, have a look at plowdown, included with plowshare.

Improve this answer
5
  • Another really perverse possibility would be that the server for some reason was configured to return 403 instead of 200 on success.
    – kasperd
    Jun 29, 2014 at 8:19
  • 1
    This gave me the clue I needed. After trying cookies, I found the referrer to be the problem (now, if only that could be spelt properly!!!)
    – starfry
    Jun 29, 2014 at 9:00
  • 2
    If it is still failing in wget try adding --auth-no-challenge. Works like magic.
    – Jonathan
    Jan 27, 2016 at 2:57
  • The --referer option solved my problem.
    – RonJohn
    Jun 2, 2020 at 7:17
  • One more possibility is that you do not have read permissions on the file on the server hosing the files. Using chmod to provide read permission resolved this issue for me.
    – Amol
    Oct 3 at 22:58
43

Tried all of the above however no luck; used dev browser tool to get user-agent string, once I added the following, success:

--user-agent="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
Improve this answer
2
18

Just want to add to the above answers that you could use the "Copy as cURL" feature present in Chrome developer tools (since v26.0) and Firebug (since v1.12). You can access this feature right-clicking the request row in the Network tab.

Improve this answer
2
  • This helped immensely, especially the tools in Chrome. When I tried in firefox, the request header after the 302 was all I could see. In Chromium I could see both and this gave me the info to solve the problem.
    – starfry
    Jun 29, 2014 at 9:02
  • 1
    @starfry You need to tick Enable persistent logs on the settings tab of Firefox's dev tools to prevent it from clearing network logs on a redirect. Chrome has a similar option. Incidentally, "Copy as cURL" has been in Firefox Nightly/Aurora/Beta for a while now, and is due next major release (31.0).
    – Bob
    Jun 29, 2014 at 14:02
5

Depending on what you're asking for, it could be a cookie. With Firefox, you can do a right-click when you're on the page in question, "View Page Info". Choose "Security" icon, and then click "View Cookies" button.

For puzzling out cookies, the Firefox "Live HTTP Headers" plug-in is essential. You can see what cookies get set, and what cookies get sent back to the web server.

wget can work with cookies, but it's totally infuriating, as it doesn't give a hint that it didn't send cookies. Your best bet is to remove all related cookies from your browser, and go through whatever initial login or page viewing sequence it takes. Look at "Live HTTP Headers" for cookies, and for any POST or GET parameters. Do the first login step with wget using "--keep-session-cookies" and "--save-cookies" options. That will give you a cookie file you can look at with a text editor. Use wget --load-cookies with the cookie file for the next steps.

Improve this answer
3
  • 1
    I tested without cookies in Firefox by opening a private browsing window and, as expected I got the 403 error. Interesting that you don't get the error in a new tab. In Chromium, a new tab returns the 403.
    – starfry
    Jun 29, 2014 at 10:04
  • 1
    Incidentally, you can use the network tab of Firefox's dev tools to inspect sent and received cookies without any addons. Ditto for Chrome/Chromium.
    – Bob
    Jun 29, 2014 at 13:57
  • @bob - yes I found that. It took me a few minutes as it wasn't something. Firebug has Copy as CURL now but it would be nice to see it the native tools too.
    – starfry
    Jun 30, 2014 at 8:28
2

Another reason this can happen is if the site requires SSL. Your browser will automatically forward from HTTP to HTTPS but curl and wget will not. So try the request with HTTPS instead of HTTP.

Improve this answer
1
  • 5
    This would end in error 301 or 302, Redirect, if I am right.
    – Jakuje
    Nov 21, 2015 at 16:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .