27

I am running Mac OS 10.11.6 El Capitan. There is a link I would like to download programmatically:

https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg

If I paste this URL into any browser (e.g. Safari) the download works perfectly.

However, if I try to download the same URL from the command line using curl, it doesn't work—the result is an empty file:

$ ls -lA
$ curl -O https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
$ ls -lA
total 0
-rw-r--r--  1 myname  staff  0 Nov  7 14:07 mysql-5.7.16-osx10.11-x86_64.dmg
$ 

Of course I can get the file through the browser, but I would like to understand why the curl command above doesn't work.

Why can't curl download this file correctly, when it is evidently present on the website and can be correctly accessed and downloaded through a graphical web browser?

  • 7
    I'd like to point out that even though techraf's answer on redirects is completely correct, other factors such as headers may cause a server to reject the curl client's request to download a file. For example, if the server has backend DDoS protection, such protection software commonly checks for viable browser headers, such as having a proper User-Agent. Additionally, some browser downloads may succeed due to session cookies (i.e if you're logged in) being only present on said browser. – Joseph A. Nov 8 '16 at 10:15
  • 6
    To troubleshoot a curl command, you can use curl -v for "verbose". It will print to standard error various info about the connection, request, and response. In this case, you would see that the response includes HTTP 302 Found (a redirect code) and a Location header with the URL to go to. Then you could man curl to find out how to tell it to follow redirects. – Nathan Long Nov 8 '16 at 17:01
57

There is a redirect on the webserver-side to the following URL: http://cdn.mysql.com//Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg. Because it's a CDN, the exact behaviour (whether you get redirected or not) might depend on your location.

curl does not follow redirects by default. To tell it to do so, add the -L argument:

curl -L -O https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg
  • 5
    A HTTPS-to-HTTP redirect? That's awful, and I'm not surprised that Curl leaves it well alone... – Toby Speight Nov 10 '16 at 17:00
  • And if this doesn't solve it? Something to do with .asp stuff maybe? – mathtick Jan 14 at 17:24
4

If the browser is able to download the file, you can inspect what the browser is doing. On google chrome you can use the following to see what’s happening.

1) [View > Developer > Developer tools > Network Tab > Headers tab]

2) Click on the download link.

3) The file link will appear on the developer tools tab.

4) Right click on the file and select Copy > Copy as cURL.

Now you have a curl link that will work. It will probably have excess parameters you can trim-away.

More details: https://lornajane.net/posts/2013/chrome-feature-copy-as-curl

0

I am going to convert one of the comments on this post to an answer.

There are many HTTP/HTTPS links that require certain headers in order to work. So this will result in a working response from a web browser but not a working response in a backend web request like curl.

I just ran into a site that required all the following headers. Failure to specify them resulted in a timeout.

  httpget.setHeader("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36");
  httpget.setHeader("Upgrade-Insecure-Requests", "1");
  httpget.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
  httpget.setHeader("Accept-Encoding", "gzip, deflate, br");
  httpget.setHeader("Accept-Language", "en-US,en;q=0.9");
  httpget.setHeader("Connection", "keep-alive");
  httpget.setHeader("Host", "www.thehost.com");

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.