1

It's not clear to me how to fetch multiple documents using curl mainly because I do not fully understand the explanation in the curl documentation

The curl documentation says:

-o, --output <file>
              Write output to <file> instead of stdout. If you are using {}  or  []  to  fetch
              multiple  documents,  you  can use '#' followed by a number in the <file> speci‐
              fier. That variable will be replaced with the current string for the  URL  being
              fetched. Like in:

               curl http://{one,two}.example.com -o "file_#1.txt"

              or use several variables like:

               curl http://{site,host}.host[1-5].com -o "#1_#2"
  • Which part of the curl command does "variable" in the above blockquote refer to?

  • The blockquote shows that you can specify multiple URLs for curl to fetch with: http://{one,two}.example.com. Does using {} cause the full URL above to expand to:

http://one.example.com
http://two.example.com 

before curl's fetch operation begins?

The curl documentation also says that:

        or you can get sequences of alphanumeric series by using [] as in:

         ftp://ftp.example.com/file[1-100].txt

         ftp://ftp.example.com/file[001-100].txt    (with leading zeros)

         ftp://ftp.example.com/file[a-z].txt
  • I'm guessing [] also performs an expansion just like {} but does the remote file name have to be exactly file1, file2, file3, ... file100? i.e. does file[1-100].txt only work for file names with a numbered sequence, e.g. will using file[1-100].txt still work if rather than file1, file2, file3,... file100 we have the files file 1, file3 file5 ... file99 at that URL?
  • Is using [] or {} the same as a shell expansion even though the curl documentation advises protecting [] or {} from shell expansion by surrounding the full URL with double quotes?:
    When using [] or {} sequences when invoked from a command  line  prompt,  you  probably
       have  to put the full URL within double quotes to avoid the shell from interfering with
       it. This also goes for other characters treated special, like for example '&', '?'  and
       '*'.
    
1

Many questions in one ...

  • Which part of the curl command does "variable" in the above blockquote refer to?
    The #1, #2 are the variables
  • Does using {} cause the full URL above to expand [...] before curl's fetch operation begins?
    Yes, that's the brace expansion like done by the shell, but protected by quotes, so curl can perform the expansion.
  • I'm guessing [] also performs an expansion just like {} but does the remote file name have to be exactly file1, file2, file3, ... file100?
    No, you can also give a step counter like file[1-99:2] for your example
  • Is using [] or {} the same as a shell expansion even though the curl documentation advises protecting [] or {} from shell expansion by surrounding the full URL with double quotes?
    The {} syntax is the same as in brace expansion, the sequence expansion is different. The reason, why you need to protect it: If the shell would already do the expansion, how could curl know what the variables should be? If the shell expands http://{one,two}.foo.com, curl sees http://one.foo.com and http://two.foo.com, but no variable. If it sees the protected original, it knows the first variable #1 to be either one or two.
  • Can you clarify your last point? Particularly what exactly replaces the variable in the example curl http://{one, two}.example.com -o "file_#1.txt". If the expansion gives http://one.foo.com and http://two.foo.com and there is only one file, which url's output is written to the file? As for what replaces the variable I don't understand what "the current string" in "That variable will be replaced with the current string for the URL being fetched" means. Does curl http://{one, two}.example.com -o "file_#1.txt" result in file_http://one.foo.com.txt or file_http://two.foo.com.txt? – MyWrathAcademia Jul 9 at 17:13
  • {one,two} will make variable #1 first take the value one and then take the value two, so http://one.example.com will be written to file_one.txt and http://two.example.com will be written to file_two.txt. (Beware that the whitespace in your {one, two} would produce two with a leading space, which is certainly not what you want!) – Philippos Jul 12 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.