2

I have a text file containing up to 100 URLs. I am able to curl the page source from them using:

cat /path/to/url.txt|xargs curl -o /path/to/output.txt

This will download the page source for all of the URLs (as i can see this happening in command line), but it will only save (in output.txt) the page source for the URL at the top of the list.

How would I go about saving page source for each URL, whether in the same text file or if necessary in individual text files.

Thanks,

3 Answers 3

1

With GNU Parallel you can get multiple URLs in parallel and you do not need to worry about the outputs getting mixed:

cat /path/to/url.txt | parallel curl > /path/to/output.txt
0
for i in $(cat urls.txt); do curl "$i" >> output.txt; done
0

with a simple list of urls on each line, this should do the job and output everything in a single file :

while read in; do xargs curl -K "$in" >> /path/to/output.txt; done < /path/to/url.txt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .