4

This question is a follow-up to How to download multiple files simultaniously with wget?

Similar to that question, I need to download many files. With the accepted answer above recommends the use of cat url-list | parallel -j8 wget {}. But this will require establishing one TCP connection and also one https/TLS negotiation per URL, which is a waste on both client and server side.

If/Since all URLs are on the same server, they could be queried in multiple GET /object.txt over a single TCP / HTTPs connection.

Question: any way to do this with wget ?

1 Answer 1

5

Wget does this by default (--no-http-keep-alive turns it off). If multiple successive URLs are on the same server, it reuses the same connection (assuming the server is willing).

You need to tell parallel to invoke wget fewer times. Pass -X to make it bunch the arguments in each invocation.

cat url-list | parallel -j8 -X wget

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .