1

I am downloading a huge list of files in parallel using xargs. I have the problem that some downloads fail after some time, so once I have run through my list, not all downloads have completed successfully, and I have to rerun the command.

cat links.txt | xargs -n 2 -P 16 wget -c -q -O

Is there a way to output the filename of successful download to a file? The only solution I could find was parsing the wget log file.

2 Answers 2

1

Use GNU Parallel instead of xargs:

cat links.txt | parallel --joblog my.log -n 2 -P 16 wget -c -q -O

Now look at the columns Exitval in my.log.

If you are going to use the exit value to retry, then GNU Parallel has options to retry a job: --retries, and --retry-failed.

0

Parsing the output of wget, or its log works. For example,

$ wget -O fixedgear.html https://www.sheldonbrown.com/fixedgear.html 2>&1 | \
    awk '/saved/ { print $6 }' | \
    sed -e 's/^.//' -e 's/.$//'
README.md

The first step in the pipe attempts to download the file, the second looks for wget to declare that the file was successfully saved and if so output it, and the last step just trims the quote marks off the filename.

Adjusting the values of --tries and --waitretry in wget may also help your initial parallel download to succeed the first time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.