53

Say I have a file with the following

bob
john
sue

Now these directly corrospond to (in this case) URL pattern such as http://example.com/persons/bob.tar, john.tar, sue.tar.

I would like to take these lines and run them through xargs. I don't know what is passed to the command being executed though. How do I access the parameter either from the prompt (say I want to simply echo each line like cat file | xargs echo $PARAM) or from a bash script.

2
  • I'm not quite following the question, sorry. In particular I'm not sure what "I don't know what is passed to the command being executed though" means Oct 29, 2010 at 2:07
  • 1
    @Michael: When you run a list through xargs it breaks it up by line and feeds each line into a command, right? How do I access that if I need to say something like cat file | xargs curl http://example.com/[PASSED FROM FILE].tar?
    – Josh K
    Oct 29, 2010 at 2:40

8 Answers 8

88

Michael's answer is right, and should sort out your problem. Running

cat file | xargs -I % curl http://example.com/persons/%.tar

will download files bob.tar john.tar. sue.tar as expected.

BUT: cat here is useless

rather use:

<file xargs -I % curl http://example.com/persons/%.tar
5
  • 1
    In my one file example it may not be ideal, however, cat xaa xab xac xad ... xargs ...
    – Josh K
    Oct 29, 2010 at 10:33
  • 13
    @Josh For some reason people tend to take unnecessary uses of cat really seriously here; I've been downvoted for it twice now Oct 29, 2010 at 12:20
  • 3
    for the record, you got a +1 from me. I think using cat is fine really, just like to add more info... :)
    – Stefan
    Oct 29, 2010 at 12:25
  • Surely it should be xargs -I % curl … (xargs option before curl and its options/arguments). At the very least xargs curl -I % (where -I % is meant as an option to xargs, not curl) is not portable. Dec 5, 2010 at 18:04
  • @Chris, nice catch, tnx. fixed.
    – Stefan
    Dec 5, 2010 at 19:59
18

I think you're asking how to insert the individual lines pulled from xargs' stdin in the middle of a command, instead of just pasting it on the end always. If so, the -I flag takes a replacement-string argument; xargs will then replace replacement-string in the command with the line read from stdin:

$ cat file | xargs -I foobar curl http://example.com/foobar.tar
2
  • Okay, how about curl http://example.com/foobar.tar > foobar.tar?
    – Josh K
    Oct 29, 2010 at 3:02
  • 2
    @Josh K: > is a shell constructs, and won't work for xargs. On the other hand, curl -o will write to a named file instead of stdout, like what wget does, so that's probably what you would like to use here.
    – ephemient
    Nov 1, 2010 at 7:23
14
$ man xargs
...
       --arg-file=file
       -a file
              Read items from file instead of standard input.  If you use this
              option,  stdin  remains unchanged when commands are run.  Other-
              wise, stdin is redirected from /dev/null.
...

You may want to set --delimiter=/-d to '\n' as well.


On the other hand, if you are just trying to turn each line in the file into a URL,

$ sed -e 's#.*#http://example.com/persons/&.tar#' file

will do, and if you want to fetch all of them, just pipe that into | wget -i.

9

another way with shell looping:

for i in `cat file`; do curl -I http://foo.com/$i; done

you can also run each iteration in the background by appending & prior to the last semicolon - for very large downloads this might be handy

4
  • 2
    You've fallen for one of the classic blunders! <-- broken link, here's a cached version: Useless Use of Cat Award.
    – Sorpigal
    Dec 6, 2010 at 17:41
  • @Sorpigal That url is broken. What is the 'Classic Blunder'? Jul 13, 2015 at 6:40
  • @starbeamrainbowlabs - useless use of cat, I've put a cached version of that link in the comment above.
    – slm
    Jul 13, 2015 at 12:24
  • @starbeamrainbowlabs: I specifically was linking to Dangerous Backticks, but the use of cat is also needless if you change to a while construct: while IFS= read i ; do curl ... ; done < file
    – Sorpigal
    Jul 14, 2015 at 20:28
6

With GNU Parallel you can do:

cat urls | parallel curl {} ">" {/}

Or:

cat persons | parallel curl http://example.com/persons/{}.tar ">" {}.tar

Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

2

while read VAR; do ... done loop is simple yet very versatile:

while read word; do wget http://example.com/persons/$word; done < file
1

Eventually, you can use as well:

xargs -n1 -a file -I % curl http://example.com/persons/%.tar
2
  • 1
    Does not using -I imply -n 1? Also, the -a option is non-standard, so you would have to specify what implementation of xargs you're using.
    – Kusalananda
    Oct 5, 2020 at 6:52
  • Using -I imply -x and -L (max-lines), but not -n ... if I am not mistaken. In xargs 4.7.0 GNU -a file is equal to --arg-file=file Oct 6, 2020 at 18:18
0

This is a more general version of Stefan:s answer but I'm using awk in the middle to prepare the exact "string" that I would like xargs to execute. And then xargs is using bash to do the actual "work".

It is a little bit overkill for this example, but it is a general solution that with some modifications can solve many problems...

cat file | awk '{print "curl http://example.com/persons/"$1".tar"}' | xargs -0 bash -c

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.