30

I am trying to download two files by the following syntax:

curl -O http://domain/path/to/{file1,file2}

The problem is that only the first file is actually saved locally, and the second was simply printed to stdout.

I do realized that if I add a -O it works just fine:

curl -OO http://domain/path/to/{file1,file2}

But isn't this impractical if the number of files grows too big? For example,

curl -O http://domain/path/to/file[1,100]

My question is, is there really no way to download multiple individual files at once with curl (without adding a correct number of -O)?

5
  • 1
    Why not using a for loop or using wget instead? Nov 15, 2015 at 12:20
  • 1
    @RuiFRibeiro (1)Yes, I can use loop. I'm just curious if curl can achieve it. (2) I'm considering not only HTTP(s), FTP protocols, e.g. SFTP.
    – Naitree
    Nov 15, 2015 at 12:25
  • I mean, being such a powerful tool, why curl just fails at this not-so-hard functionality?
    – Naitree
    Nov 15, 2015 at 12:29
  • 2
    Just to make clear that the {} and [] (and also * and ?) syntax is shell globbing/expansion, curl never gets to see them, it is the same as curl url1 url2
    – vonbrand
    Nov 15, 2015 at 13:36
  • wget does not handle SSL
    – MikeW
    Aug 15, 2017 at 12:33

4 Answers 4

36

I realize that there is already an accepted answer to this question but I feel I should point out that there actually is a way to do this.

The --remote-name-all option tells curl to behave as if you used -O or --remote-name for each file.

https://curl.haxx.se/docs/manpage.html#--remote-name-all

This option has been available since version 7.19.0

https://curl.haxx.se/changes.html#7_19_0

16

This has been implemented in curl 7.19.0. See @Besworks answer.

According to the man page there is no way to keep the original file name except using multiple Os. Alternatively you could use your own file names:

curl http://{one,two}.site.example -o "file_#1.txt"

resulting in http://one.site.example being saved to file_one.txt and http://two.site.example being saved to file_two.txt.

Multiple variables even work. Like:

curl http://{site,host}.host[1-5].example -o "#1_#2"

resulting in http://site.host1.example being saved to site_1, http://host.host1.example being saved to host_1 and so on.

1
  • 1
    Or you could do something like curl -$(printf 'O%.0s' {1..3}) http://domain/path/to/{file1,file2,file3}
    – Max Coplan
    Sep 11, 2019 at 17:02
6

The problem here is how the shell (likely BASH) is interpreting the curl command.

Basically, it is looking at

curl -O http://domain/path/to/{file1,file2}

and expanding it to:

curl -O http://domain/path/to/file1 http://domain/path/to/file2

This is a problem because the flag, -O, is applied only to the first instance and not to any that follow it.

This can be fixed by double quoting the url:

curl -O "http://domain/path/to/{file1,file2}"

The curl team have acknowledged this on GitHub and PRs to both the man-page and the official curl manual have been submitted so they should reflect this potential globbing behaviour in some shells in the future.

If you'd like an easy test case to see this failing and how to fix it then try downloading two zips of .txt files of Moby Dick from Project Gutenberg.

Running:

curl -O https://www.gutenberg.org/files/2701/2701-{0,h}.zip

will provide the following response:

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 500k 100 500k 0 0 380k 0 0:00:01 0:00:01 --:--:-- 380k Warning: Binary output can mess up your terminal. Use "--output -" to tell Warning: curl to output it to your terminal anyway, or consider "--output Warning: <FILE>" to save to a file.

(If the files you were pulling were raw text then it would just dump to standard out, likely your screen, and that would be mean to do for Moby Dick and that's why the .zip files are being used instead.)

Running:

curl -O "https://www.gutenberg.org/files/2701/2701-{0,h}.zip"

will give something like this output instead:

[1/2]: https://www.gutenberg.org/files/2701/2701-0.zip --> 2701-0.zip --_curl_--https://www.gutenberg.org/files/2701/2701-0.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 500k 100 500k 0 0 407k 0 0:00:01 0:00:01 --:--:-- 407k

[2/2]: https://www.gutenberg.org/files/2701/2701-h.zip --> 2701-h.zip --_curl_--https://www.gutenberg.org/files/2701/2701-h.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 522k 100 522k 0 0 1000k 0 --:--:-- --:--:-- --:--:-- 1000k

Note that this problem only arises when { } are used for globbing. If only [ ] are used then the double quotes are not needed.

Incidentally, if wget is installed it can likely handle the globbing without the need for quotes.

Running:

wget http://domain/path/to/{file1,file2}

would pull down both files (if they actually existed).

Running:

wget https://www.gutenberg.org/files/2701/2701-{0,h}.zip

will pull down the Moby Dick zip files as per the double-quoted curl example, above.

2

There is an alternative way to download multiple files with curl:

urls="firstUrl secondUrl thirdUrl" 
for url in $urls
do
   curl -O "$url"
done

Note: the mandatory space is used to delimit different URLs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.