Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Say I have a file with the following

bob
john
sue

Now these directly corrospond to (in this case) URL pattern such as http://example.com/persons/bob.tar, john.tar, sue.tar.

I would like to take these lines and run them through xargs. I don't know what is passed to the command being executed though. How do I access the parameter either from the prompt (say I want to simply echo each line like cat file | xargs echo $PARAM) or from a bash script.

share|improve this question
    
I'm not quite following the question, sorry. In particular I'm not sure what "I don't know what is passed to the command being executed though" means –  Michael Mrozek Oct 29 '10 at 2:07
    
@Michael: When you run a list through xargs it breaks it up by line and feeds each line into a command, right? How do I access that if I need to say something like cat file | xargs curl http://example.com/[PASSED FROM FILE].tar? –  Josh K Oct 29 '10 at 2:40
add comment

7 Answers

Michael's answer is right, and should sort out your problem. Running

cat file | xargs -I % curl http://example.com/persons/%.tar

will download files bob.tar john.tar. sue.tar as expected.

BUT: Cat is Useless

rather use:

<file xargs -I % curl http://example.com/persons/%.tar
share|improve this answer
1  
In my one file example it may not be ideal, however, cat xaa xab xac xad ... xargs ... –  Josh K Oct 29 '10 at 10:33
3  
@Josh For some reason people tend to take unnecessary uses of cat really seriously here; I've been downvoted for it twice now –  Michael Mrozek Oct 29 '10 at 12:20
    
for the record, you got a +1 from me. I think using cat is fine really, just like to add more info... :) –  NixNinja Oct 29 '10 at 12:25
    
Surely it should be xargs -I % curl … (xargs option before curl and its options/arguments). At the very least xargs curl -I % (where -I % is meant as an option to xargs, not curl) is not portable. –  Chris Johnsen Dec 5 '10 at 18:04
    
@Chris, nice catch, tnx. fixed. –  NixNinja Dec 5 '10 at 19:59
add comment

I think you're asking how to insert the individual lines pulled from xargs' stdin in the middle of a command, instead of just pasting it on the end always. If so, the -I flag takes a replacement-string argument; xargs will then replace replacement-string in the command with the line read from stdin:

$ cat file | xargs -I foobar curl http://example.com/foobar.tar
share|improve this answer
    
Okay, how about curl http://example.com/foobar.tar > foobar.tar? –  Josh K Oct 29 '10 at 3:02
    
@Josh K: > is a shell constructs, and won't work for xargs. On the other hand, curl -o will write to a named file instead of stdout, like what wget does, so that's probably what you would like to use here. –  ephemient Nov 1 '10 at 7:23
add comment
$ man xargs
...
       --arg-file=file
       -a file
              Read items from file instead of standard input.  If you use this
              option,  stdin  remains unchanged when commands are run.  Other-
              wise, stdin is redirected from /dev/null.
...

You may want to set --delimiter=/-d to '\n' as well.


On the other hand, if you are just trying to turn each line in the file into a URL,

$ sed -e 's#.*#http://example.com/persons/&.tar#' file

will do, and if you want to fetch all of them, just pipe that into | wget -i.

share|improve this answer
add comment

another way with shell looping:

for i in `cat file`; do curl -I http://foo.com/$i; done

you can also run each iteration in the background by appending & prior to the last semicolon - for very large downloads this might be handy

share|improve this answer
    
+1 I like the bash for loop –  Johan Dec 6 '10 at 16:46
2  
You've fallen for one of the classic blunders! –  Sorpigal Dec 6 '10 at 17:41
add comment

With GNU Parallel you can do:

cat urls | parallel curl {} ">" {/}

Or:

cat persons | parallel curl http://example.com/persons/{}.tar ">" {}.tar

Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

share|improve this answer
add comment

This is a more general version of Stefan:s answer but I'm using awk in the middle to prepare the exact "string" that I would like xargs to execute. And then xargs is using bash to do the actual "work".

It is a little bit overkill for this example, but it is a general solution that with some modifications can solve many problems...

cat file | awk '{print "curl http://example.com/persons/"$1".tar"}' | xargs -0 bash -c
share|improve this answer
add comment

while read VAR; do ... done loop is simple yet very versatile:

while read word; do wget http://example.com/persons/$word; done < file
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.