How to speed up curl?

Question

I wrote a script on my Debian machine to read a file of URLs and output the HTTP Status code for each one:

...
while read url
do
    urlstatus=$(curl -H 'Cache-Control: no-cache' -o /dev/null --silent --head --insecure --write-out '%{http_code} ; %{redirect_url}' "$url")
    ...
done < $1

With a file of 100 URLs it's OK, but with a huge file with more than 1000 URLs, my script becomes very slow!

I've tried several 'tips' found on SO : set KeepAlive with -H 'Keep-Alive: 500', define GoogleBot as User Agent with --user-agent 'Googlebot/2.1 (+http://www.google.com/bot.html)' and define a connection time out with --connect-timeout 1 but nothing!

Answer 1

If you want to run them in parallel you can use GNU parallel:

#!/usr/bin/env bash

file=$1

status_func () {
    local _url="${1:-/dev/stdin}"
    local _urlstatus
    _urlstatus=$(curl -H 'Cache-Control: no-cache' -o /dev/null --silent --head --insecure --write-out '%{http_code} ; %{redirect_url}' "$_url")
    printf 'Status: %s\n' "$_urlstatus"
}

export -f status_func

cat "$file" | parallel -j20 status_func

The -j option will set the maximum number of simultaneous jobs, the default is the number of threads on your system but you may need to tweak that number for optimal results.

Note you will have to add your commands into a function (or alternatively a separate script altogether) and export that function so that parallel knows about it.

If you were to put your commands into a standalone script than you could just run it from the command line like:

cat "$file" | parallel -j20 ./my_script.sh

Note this script should expect only a single URL sent via stdin.

---

Alternatively if you cannot use GNU parallel but still want to run multiple jobs at once you could just execute them as background jobs but will most likely want to limit the number of simultaneous processes. That can be accomplished with the following:

#!/usr/bin/env bash

file=$1

max_bg_procs () {
    local max_number=$((0 + ${1:-0}))
    while :; do
        local current_number=$(jobs -pr | wc -l)
        if [[ $current_number -lt $max_number ]]; then
            break
        fi
        sleep 2
    done
}

status_func () {
    local _url="${1:-/dev/stdin}"
    local _urlstatus
    _urlstatus=$(curl -H 'Cache-Control: no-cache' -o /dev/null --silent --head --insecure --write-out '%{http_code} ; %{redirect_url}' "$_url")
    printf 'Status: %s\n' "$_urlstatus"
}

while IFS= read -rd url; do
    max_bg_procs 20
    status_func "$url" &
done < "$file"

Answer 2

Use xargs -P0

seq 100 | xargs -I@ echo 'https://example.com?@' > urls.txt
cat urls.txt | xargs -P0 -n1 curl

How fast?

Without -P0:

time sh -c 'cat urls.txt | xargs -n1 curl'

real    0m 59.79s
user    0m 1.64s
sys     0m 0.99s

With -P0:

time sh -c 'cat urls.txt | xargs -P0 -n1 curl'

real    0m 0.76s
user    0m 1.26s
sys     0m 0.82s

Quoted from man xargs:

-P maxprocs

Parallel mode: run at most maxprocs invocations of utility at once. If maxprocs is set to 0, xargs will run as many processes as possible.

Answer 3

Use -Z

Another potential solution depending on your use case is the -Z flag. This is useful when providing multiple URLs to curl or when having curl automate things like pagination by specifying a curl specific syntax to work on a range.

The documentation for the -Z flag is https://curl.se/docs/manpage.html#-Z

The documentation for curl specific range syntax is in https://curl.se/docs/manpage.html and ^F 'Globbing'.

curl -Z $(cat urls.txt | xargs)