57

Say I have a file with the following

bob
john
sue

Now these directly corrospond to (in this case) URL pattern such as http://example.com/persons/bob.tar, john.tar, sue.tar.

I would like to take these lines and run them through xargs. I don't know what is passed to the command being executed though. How do I access the parameter either from the prompt (say I want to simply echo each line like cat file | xargs echo $PARAM) or from a bash script.

2
  • I'm not quite following the question, sorry. In particular I'm not sure what "I don't know what is passed to the command being executed though" means Commented Oct 29, 2010 at 2:07
  • 1
    @Michael: When you run a list through xargs it breaks it up by line and feeds each line into a command, right? How do I access that if I need to say something like cat file | xargs curl http://example.com/[PASSED FROM FILE].tar?
    – Josh K
    Commented Oct 29, 2010 at 2:40

8 Answers 8

94

Michael's answer is right, and should sort out your problem. Running

cat file | xargs -I % curl http://example.com/persons/%.tar

will download files bob.tar john.tar. sue.tar as expected.

BUT: cat here is useless

rather use:

<file xargs -I % curl http://example.com/persons/%.tar
5
  • 1
    In my one file example it may not be ideal, however, cat xaa xab xac xad ... xargs ...
    – Josh K
    Commented Oct 29, 2010 at 10:33
  • 15
    @Josh For some reason people tend to take unnecessary uses of cat really seriously here; I've been downvoted for it twice now Commented Oct 29, 2010 at 12:20
  • 4
    for the record, you got a +1 from me. I think using cat is fine really, just like to add more info... :)
    – Stefan
    Commented Oct 29, 2010 at 12:25
  • Surely it should be xargs -I % curl … (xargs option before curl and its options/arguments). At the very least xargs curl -I % (where -I % is meant as an option to xargs, not curl) is not portable. Commented Dec 5, 2010 at 18:04
  • @Chris, nice catch, tnx. fixed.
    – Stefan
    Commented Dec 5, 2010 at 19:59
19

I think you're asking how to insert the individual lines pulled from xargs' stdin in the middle of a command, instead of just pasting it on the end always. If so, the -I flag takes a replacement-string argument; xargs will then replace replacement-string in the command with the line read from stdin:

$ cat file | xargs -I foobar curl http://example.com/foobar.tar
2
  • Okay, how about curl http://example.com/foobar.tar > foobar.tar?
    – Josh K
    Commented Oct 29, 2010 at 3:02
  • 3
    @Josh K: > is a shell constructs, and won't work for xargs. On the other hand, curl -o will write to a named file instead of stdout, like what wget does, so that's probably what you would like to use here.
    – ephemient
    Commented Nov 1, 2010 at 7:23
16
$ man xargs
...
       --arg-file=file
       -a file
              Read items from file instead of standard input.  If you use this
              option,  stdin  remains unchanged when commands are run.  Other-
              wise, stdin is redirected from /dev/null.
...

You may want to set --delimiter=/-d to '\n' as well.


On the other hand, if you are just trying to turn each line in the file into a URL,

$ sed -e 's#.*#http://example.com/persons/&.tar#' file

will do, and if you want to fetch all of them, just pipe that into | wget -i.

9

another way with shell looping:

for i in `cat file`; do curl -I http://foo.com/$i; done

you can also run each iteration in the background by appending & prior to the last semicolon - for very large downloads this might be handy

4
  • 2
    You've fallen for one of the classic blunders! <-- broken link, here's a cached version: Useless Use of Cat Award.
    – Sorpigal
    Commented Dec 6, 2010 at 17:41
  • @Sorpigal That url is broken. What is the 'Classic Blunder'? Commented Jul 13, 2015 at 6:40
  • @starbeamrainbowlabs - useless use of cat, I've put a cached version of that link in the comment above.
    – slm
    Commented Jul 13, 2015 at 12:24
  • @starbeamrainbowlabs: I specifically was linking to Dangerous Backticks, but the use of cat is also needless if you change to a while construct: while IFS= read i ; do curl ... ; done < file
    – Sorpigal
    Commented Jul 14, 2015 at 20:28
6

With GNU Parallel you can do:

cat urls | parallel curl {} ">" {/}

Or:

cat persons | parallel curl http://example.com/persons/{}.tar ">" {}.tar

Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

2

while read VAR; do ... done loop is simple yet very versatile:

while read word; do wget http://example.com/persons/$word; done < file
2

Eventually, you can use as well:

xargs -n1 -a file -I % curl http://example.com/persons/%.tar
2
  • 1
    Does not using -I imply -n 1? Also, the -a option is non-standard, so you would have to specify what implementation of xargs you're using.
    – Kusalananda
    Commented Oct 5, 2020 at 6:52
  • Using -I imply -x and -L (max-lines), but not -n ... if I am not mistaken. In xargs 4.7.0 GNU -a file is equal to --arg-file=file Commented Oct 6, 2020 at 18:18
0

This is a more general version of Stefan:s answer but I'm using awk in the middle to prepare the exact "string" that I would like xargs to execute. And then xargs is using bash to do the actual "work".

It is a little bit overkill for this example, but it is a general solution that with some modifications can solve many problems...

cat file | awk '{print "curl http://example.com/persons/"$1".tar"}' | xargs -0 bash -c

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .