I have a text file containing up to 100 URLs. I am able to curl the page source from them using:

cat /path/to/url.txt|xargs curl -o /path/to/output.txt

This will download the page source for all of the URLs (as i can see this happening in command line), but it will only save (in output.txt) the page source for the URL at the top of the list.

How would I go about saving page source for each URL, whether in the same text file or if necessary in individual text files.



With GNU Parallel you can get multiple URLs in parallel and you do not need to worry about the outputs getting mixed:

cat /path/to/url.txt | parallel curl > /path/to/output.txt
| improve this answer | |
for i in $(cat urls.txt); do curl "$i" >> output.txt; done
| improve this answer | |

with a simple list of urls on each line, this should do the job and output everything in a single file :

while read in; do xargs curl -K "$in" >> /path/to/output.txt; done < /path/to/url.txt
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.