2

I have a file which has many links. As an example here is my file:

link1
link2
link3
link4
link5
link6
link7
link8
link9
link10
link11
link12
link13
link14
link15
link16

I want to run a particular command on a single terminal which runs 1st 4 links simultaneously. For example the command I want to run is

wget link1
wget link2

and etc.. but I want to run first 4 links simultaneously. Then if one of the link is finished downloading then the next link(which is link5) should be automatically submitted for download and so on.

I am looking for a way where I am not downloading the links one by one(uses more time), neither I have to open multiple terminals to submit multiple individual links as separate commands. Any help would be really appreciated.

1
  • Your should not have to use separate terminals. Adding a & to the end of a command will give you back the prompt. See shell job control. However using gnu parallel is better (automated). Dec 12 '19 at 22:18
5

You could use GNU parallel:

parallel --retries 5 -j4 -a file.txt wget {}

This will execute 4 jobs at a time and read input from file.txt. If a job fails it will be retried up to 20 times before moving on to the next job.

6
  • You are missing the part Then if one of the link is finished downloading then the next link(which is link5) should be automatically submitted for download and so on Dec 12 '19 at 22:07
  • @guillermochamorro: That is what -j4 does, although it doesn't actually check that it finished successfully
    – jesse_b
    Dec 12 '19 at 22:10
  • 1
    Add --retries 20 to parallel to retry if a download fails.
    – Ole Tange
    Dec 14 '19 at 10:34
  • Thanks @OleTange
    – jesse_b
    Dec 14 '19 at 13:56
  • wget retries 20 times by default.
    – Kusalananda
    Dec 14 '19 at 16:11
2

With an xargs that implements the -P option for keeping multiple jobs running in parallel (most common implementations do):

xargs -I {} -P 4 wget --quiet {} <file.txt

Doing it without xargs or GNU parallel, but using bash:

#!/bin/bash

jobs=0
while read -r url; do
    if [ "$jobs" -ge 4 ]; then
        wait -n
        jobs=$(( jobs - 1 ))
    fi

    wget --quiet "$url" &
    jobs=$(( jobs + 1 ))
done <file.txt

wait

This starts wget background tasks as fast as possible until there has been four such jobs started. Then it waits for any of these to end with wait -n before starting the next one. The jobs variable holds the number of currently running wget jobs.

At the end, the single wait call will block until all jobs have finished.

It's the wait -n that makes this a bash script rather than a plain /bin/sh script.

0

While all the answers using parallel and xargs work fine, allow me to introduce you to GNU Wget2. It's the next version of Wget, and while it is still in alpha-mode, it is a drop-in replacement for most usages of Wget.

Wget2 supports multithreaded downloads, so you can simply give it the file and how many downloads you want in parallel, let Wget2 handle the rest for you.

Source: Am maintainer for both GNU Wget and GNU Wget2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.