Why does wget work and curl fail with a specific REST URL?

Question

In answering a question on another site, I came across a strange discrepancy between curl and wget when trying to fetch from this URL: https://www.uniprot.org/uniprot/A2Z669.fasta

For some reason, curl just silently fails to download, while wget correctly fetches the file A2Z669.fasta:

$ ls -la
total 300
drwxr-xr-x   2 terdon terdon 266240 Dec 11 12:22 .
drwxr-xr-x 202 terdon terdon  32768 Dec 10 17:31 ..

$ curl https://www.uniprot.org/uniprot/A2Z669.fasta
$ ls -la
total 300
drwxr-xr-x   2 terdon terdon 266240 Dec 11 12:22 .
drwxr-xr-x 202 terdon terdon  32768 Dec 10 17:31 ..

Explicitly setting an output file doesn't help, that just creates an empty file:

$ curl -o file "https://www.uniprot.org/uniprot/A2Z669.fasta"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

$ ls -la
total 300
drwxr-xr-x   2 terdon terdon 266240 Dec 11 12:25 .
drwxr-xr-x 202 terdon terdon  32768 Dec 10 17:31 ..
-rw-r--r--   1 terdon terdon      0 Dec 11 12:25 file
$ cat file
$ 

And yet, wget works just fine:

$ wget https://www.uniprot.org/uniprot/A2Z669.fasta
--2023-12-11 12:24:42--  https://www.uniprot.org/uniprot/A2Z669.fasta
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving www.uniprot.org (www.uniprot.org)... 193.62.193.81
Connecting to www.uniprot.org (www.uniprot.org)|193.62.193.81|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://rest.uniprot.org/uniprot/A2Z669.fasta [following]
--2023-12-11 12:24:42--  https://rest.uniprot.org/uniprot/A2Z669.fasta
Resolving rest.uniprot.org (rest.uniprot.org)... 193.62.193.81
Connecting to rest.uniprot.org (rest.uniprot.org)|193.62.193.81|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://rest.uniprot.org/uniprotkb/A2Z669.fasta [following]
--2023-12-11 12:24:43--  https://rest.uniprot.org/uniprotkb/A2Z669.fasta
Reusing existing connection to rest.uniprot.org:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘A2Z669.fasta’

A2Z669.fasta            [ <=>               ]     314  --.-KB/s    in 0s      

2023-12-11 12:24:43 (6.65 MB/s) - ‘A2Z669.fasta’ saved [314]

$ ls -la
total 304
drwxr-xr-x   2 terdon terdon 266240 Dec 11 12:24 .
drwxr-xr-x 202 terdon terdon  32768 Dec 10 17:31 ..
-rw-r--r--   1 terdon terdon    314 Dec 11 12:24 A2Z669.fasta

It doesn't seem to be specific to that particular file either. I tried another URL from the same REST API (https://www.uniprot.org/uniprot/P05067.fasta) and got the same behavior.

I am running this on an Arch system with:

$ wget --version | head -n1
GNU Wget 1.21.4 built on linux-gnu.

$ curl --version | head -n1
curl 8.4.0 (x86_64-pc-linux-gnu) libcurl/8.4.0 OpenSSL/3.1.4 zlib/1.3 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.11.0 nghttp2/1.58.0

What's going on here? What would make wget work when curl fails?

Answer 1

wget follows redirections by default, curl doesn’t. If you add -L, curl works fine:

curl -OL https://www.uniprot.org/uniprot/A2Z669.fasta

(-O tells curl to output to a file instead of standard output, matching wget’s default behaviour.)