12

I have to set ACL on a multiple files. I have downloaded a list of object stored using below commands.

C:\Users\Gshrivastava\Downloads\curl_748_0>curl  -o urlname.csv -i -k -H "Authorization: HCP bXFl:29def7dbc8892a9389ebc7a5210dd844" -H "Content-Type: application/xml" -H "Accept:application/xml" -d @mqe.xml "http://tenant.hcp3.hdsblr.com/query?prettyprint

I have then sorted url names into a text file .

ns.tenant.hcp3.hdsblr.com/rest/pic/Cat/images.jpg 
ns.tenant.hcp3.hdsblr.com/rest/pic/Cat/6.png 
ns.tenant.hcp3.hdsblr.com/rest/pic/landscape/9.png 
ns.tenant.hcp3.hdsblr.com/rest/pic/landscape/5.png  

content of text file >

Now I want to use this file as argument or variable so that all the filenames are set with ACL.

curl.exe -k http://ns.tenant.hcp3.hdsblr.com/rest/ACL/filename.ext/?type=acl -i -H "Authorization: HCP YWRtaW4=:29def7dbc8892a9389ebc7a5210dd844" -T acl.xml 
2
  • Looks like you're on Windows? Try the SuperUser SE, you're on the Linux site here :)
    – Tim Malone
    May 9 '16 at 10:12
  • 3
    I hope this isn't your real authorization cookie here! May 9 '16 at 21:09
11

If I understand correctly, you have a file containing a list of URLs (one per line), and you want to pass those URLs to CURL.

There are two main ways to do that: with xargs, or with command substitution. With xargs:

xargs <urls.txt curl …

With command substitution:

curl … $(cat urls.txt)

Both methods mangle some special characters, but given what characters are valid in URLs, this shouldn't be an issue, except that with xargs, single quotes (') need to be encoded as %27. Alternatively, use xargs -l.

Note that since this is a Unix site, I'm assuming that you're running a Unix variant and invoking these commands from a Unix shell such as bash. Given that you're running curl.exe, you appear to be using Windows. If you're going to use Unix tools, I recommend that you do so from a Unix shell such as bash or zsh; Windows does not come with xargs any more than it comes with curl, and cmd does not have command substitution (at least not in the same form). There is probably a way to do this with Windows tools, but I don't know what it is and it's off-topic here.

Also, if you're using Unix tools under Windows, take care that your list of URLs uses Unix line endings (LF only), not Windows line endings (CR+LF). Unix tools expect a line to end with LF and treat CR as an ordinary character. For more information, see Directories are listed twice and many other questions on this site.

1
3

curl has the -K options where you can pass multiple urls, reads from a file that has this format:

url = url1
# Uncomment if you want to download the file 
# output = "file1"
# Uncomment if your sysadmin only allows well known User Agent 
# user-agent = "Mozilla/5.0"

Also you can use xargs (wget - i style)

$ xargs -a urls.txt -I{} curl -# -O {}

urls.txt format:

https://download.libsodium.org/libsodium/releases/LATEST.tar.gz
https://download.libsodium.org/libsodium/releases/libsodium-1.0.16.tar.gz.sig

Using HEREDOC

curl -# -K - <<URL
    url = "https://download.libsodium.org/libsodium/releases/libsodium-1.0.16.tar.gz"
    output = "libsodium-1.0.16.tar.gz"

    url = "https://download.libsodium.org/libsodium/releases/libsodium-1.0.16.tar.gz.sig"
   output="libsodium-1.0.16.tar.gz.sig"
URL

Reference

5
  • > reads from a file that has this format It'd be nice if you could give an example with more than one set of inputs. It isn't 100% clear how curl delimits the different sets of options...is it anything between a 'url=' and the next one (or eof)? Apr 1 at 9:17
  • The -Z option can be combined with -K - to read lots of URLs from a file and download them a lot more quickly than running multiple curl commands...needs curl v7.70.0 or greater. Apr 8 at 12:25
  • @MaxWaterman every url config should be in its own line that is enough.
    – Sr. Libre
    Apr 9 at 15:21
  • @Max Waterman Thank you for the -Z option very useful for mass file download.
    – Sr. Libre
    Apr 9 at 15:22
  • > every url config should be in its own line that is enough < yeah, but what about the other command line options - do they need to be duplicated, or specified separately (if they're different for each URL), or what? Apr 12 at 1:11
3

Gilles' solution didn't work for me, so I created a loop solution. One curl call per line.

# for looking at the commands first.
$ cat urls.txt | while read f; do echo curl "${f}" -O; done;

# remove echo when you feel safe.
$ cat urls.txt | while read f; do curl "${f}" -O; done;

UPDATE: As requested, an attempt to improve performance with a single curl call:

$ cat urls.txt | while read f; do echo -e "url = \"${f}\"\n-O\n" >> config; done;
$ curl -K config
1
  • 1
    It's a bit inefficient to do one curl per line, imo. Can you make one that uses the '-K -' option? Apr 1 at 9:15
0

It appears you're on Windows and are using a CMD Prompt. The above example assume you have a linux environment. But in case you're wanting to do this in your CMD Prompt, create a batch file (such as mycurlscript.bat) and put this in it:

@echo off
for /F "tokens=*" %%A in (urls.txt) do curl.exe -k %%A -i -H "Authorization: HCP YWRtaW4=:29def7dbc8892a9389ebc7a5210dd844" -T acl.xml

Then save your list of urls you want to download into a file named urls.txt. Put these into the same directory as your acl.xml file and from your CMD Prompt run: mycurlscript.bat

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.