11

In answering a question on another site, I came across a strange discrepancy between curl and wget when trying to fetch from this URL: https://www.uniprot.org/uniprot/A2Z669.fasta

For some reason, curl just silently fails to download, while wget correctly fetches the file A2Z669.fasta:

$ ls -la
total 300
drwxr-xr-x   2 terdon terdon 266240 Dec 11 12:22 .
drwxr-xr-x 202 terdon terdon  32768 Dec 10 17:31 ..

$ curl https://www.uniprot.org/uniprot/A2Z669.fasta
$ ls -la
total 300
drwxr-xr-x   2 terdon terdon 266240 Dec 11 12:22 .
drwxr-xr-x 202 terdon terdon  32768 Dec 10 17:31 ..

Explicitly setting an output file doesn't help, that just creates an empty file:

$ curl -o file "https://www.uniprot.org/uniprot/A2Z669.fasta"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

$ ls -la
total 300
drwxr-xr-x   2 terdon terdon 266240 Dec 11 12:25 .
drwxr-xr-x 202 terdon terdon  32768 Dec 10 17:31 ..
-rw-r--r--   1 terdon terdon      0 Dec 11 12:25 file
$ cat file
$ 

And yet, wget works just fine:

$ wget https://www.uniprot.org/uniprot/A2Z669.fasta
--2023-12-11 12:24:42--  https://www.uniprot.org/uniprot/A2Z669.fasta
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving www.uniprot.org (www.uniprot.org)... 193.62.193.81
Connecting to www.uniprot.org (www.uniprot.org)|193.62.193.81|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://rest.uniprot.org/uniprot/A2Z669.fasta [following]
--2023-12-11 12:24:42--  https://rest.uniprot.org/uniprot/A2Z669.fasta
Resolving rest.uniprot.org (rest.uniprot.org)... 193.62.193.81
Connecting to rest.uniprot.org (rest.uniprot.org)|193.62.193.81|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://rest.uniprot.org/uniprotkb/A2Z669.fasta [following]
--2023-12-11 12:24:43--  https://rest.uniprot.org/uniprotkb/A2Z669.fasta
Reusing existing connection to rest.uniprot.org:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘A2Z669.fasta’

A2Z669.fasta            [ <=>               ]     314  --.-KB/s    in 0s      

2023-12-11 12:24:43 (6.65 MB/s) - ‘A2Z669.fasta’ saved [314]

$ ls -la
total 304
drwxr-xr-x   2 terdon terdon 266240 Dec 11 12:24 .
drwxr-xr-x 202 terdon terdon  32768 Dec 10 17:31 ..
-rw-r--r--   1 terdon terdon    314 Dec 11 12:24 A2Z669.fasta

It doesn't seem to be specific to that particular file either. I tried another URL from the same REST API (https://www.uniprot.org/uniprot/P05067.fasta) and got the same behavior.

I am running this on an Arch system with:

$ wget --version | head -n1
GNU Wget 1.21.4 built on linux-gnu.

$ curl --version | head -n1
curl 8.4.0 (x86_64-pc-linux-gnu) libcurl/8.4.0 OpenSSL/3.1.4 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.58.0

What's going on here? What would make wget work when curl fails?

2
  • Cross-site dupe: askubuntu.com/q/1036484
    – heemayl
    Dec 12, 2023 at 14:30
  • I always use curl -vv to "debug" issues with some url/http call. You should see the redirect if you add that option.
    – Bakuriu
    Dec 12, 2023 at 21:41

1 Answer 1

26

wget follows redirections by default, curl doesn’t. If you add -L, curl works fine:

curl -OL https://www.uniprot.org/uniprot/A2Z669.fasta

(-O tells curl to output to a file instead of standard output, matching wget’s default behaviour.)

9
  • 1
    Ah! Of course! I'm an idiot: that redirects to rest.uniprot.org/uniprotkb/A2Z669.fasta and that one I can get with curl.
    – terdon
    Dec 11, 2023 at 10:47
  • 9
    Not an idiot ;-). curl’s handling of redirections is easy to forget. Dec 11, 2023 at 10:49
  • 13
    In my mind, I have classified wget as a "command line browser" and curl as a "shell API to HTTP". I find, thinking about it this way helps me to understand why certain defaults are chosen. E.g. the default of wget to download to a file whose name matches the URI vs. curl streaming to stdout. Dec 11, 2023 at 11:33
  • 2
    @JörgWMittag I tend to think along similar lines — wget as a command-line download tool (with its handy “mirror” features), and curl as a command-line HTTP tool. Dec 12, 2023 at 8:27
  • 4
    Easy to forget because it's an annoying default. It would make far more sense to require an option to stop at a redirect.
    – Barmar
    Dec 12, 2023 at 15:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .