3

I have about 10 links in file links.txt. I download them through

cat links.txt | xargs -i --max-procs=4 bash -c "wget {}"

I whould like do it repeatedly like

while :
do
   cat links.txt | xargs -i --max-procs=4 bash -c "wget {}"
done

Before the next iteration of while loop it waits all subprocesses to be finished. But I would like to continue downloading (take url from beginning of the file) when there is an available thread. How to do that? May be I can make infinite pipe of strings?

1
  • 5
    I would try to loop only the cat: while :; do cat links.txt; done | xargs -i --max-procs=4 bash -c "wget {}"
    – manatwork
    Apr 25, 2014 at 9:34

1 Answer 1

3

If your yes accepts a string argument:

yes links.txt | xargs cat | xargs -n1 --max-procs=4 wget

No point in running one cat perl links.txt or one bash per link (unless you want the lines of links.txt to be interpreted as shell code like http://$host$path or http://evil;rm -rf /).

If the file is small enough, you can also do:

yes -- "$(cat links.txt)" | xargs -n1 -P4 wget

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .