233

So far I couldn't find anything really, but is it true that curl doesn't really time out at all?

 user@host:~# curl http://localhost/testdir/image.jpg

I'm asking because I'm redirecting any request for images in testdir to a separate Apache module which generates those pictures on the fly. It can take up to 15 minutes before the picture is actually ready and delivered to the requesting client.

Will curl always wait (or is it depending on configuration) or is there any sort of timeout?

  • 3
    I would expect curl to have a connection timeout (if nothing else, the OS and its TCP/IP stack almost certainly does), but it might not have a read timeout once the connection has been established. – a CVn Oct 11 '13 at 11:40
318

Yes.

Timeout parameters

curl has two options: --connect-timeout and --max-time.

Quoting from the manpage:

--connect-timeout <seconds>
    Maximum  time  in  seconds  that you allow the connection to the
    server to take.  This only limits  the  connection  phase,  once
    curl has connected this option is of no more use.  Since 7.32.0,
    this option accepts decimal values, but the actual timeout  will
    decrease in accuracy as the specified timeout increases in deci‐
    mal precision. See also the -m, --max-time option.

    If this option is used several times, the last one will be used.

and:

-m, --max-time <seconds>
    Maximum  time  in  seconds that you allow the whole operation to
    take.  This is useful for preventing your batch jobs from  hang‐
    ing  for  hours due to slow networks or links going down.  Since
    7.32.0, this option accepts decimal values, but the actual time‐
    out will decrease in accuracy as the specified timeout increases
    in decimal precision.  See also the --connect-timeout option.

    If this option is used several times, the last one will be used.

Defaults

Here (on Debian) it stops trying to connect after 2 minutes, regardless of the time specified with --connect-timeout and although the default connect timeout value seems to be 5 minutes according to the DEFAULT_CONNECT_TIMEOUT macro in lib/connect.h.

A default value for --max-time doesn't seem to exist, making curl wait forever for a response if the initial connect succeeds.

What to use?

You are probably interested in the latter option, --max-time. For your case set it to 900 (15 minutes).

Specifying option --connect-timeout to something like 60 (one minute) might also be a good idea. Otherwise curl will try to connect again and again, apparently using some backoff algorithm.

  • 2
    thanks for that! --max-time doesn't say anything about default values, so i guess it has no and therefore has no timeout except the connect-timeout by default...? – Preexo Oct 11 '13 at 13:30
  • 3
    Yes, if the connect succeeds then curl seems to wait forever for a response. – scai Oct 11 '13 at 13:38
  • Note that both maxtime is a problem if the response is a big download that takes longer than 'maxtime'. – user92979 Mar 15 '18 at 2:22
  • The 2 minute timeout also smells to me like a server time out. Just had same problem with the http server of a Node.js app which has a default timeout of 2 minutes. To increase it, see the HTTP.server.setTimeout(). – Thalis K. Apr 2 '18 at 17:56
16

There is timelimit: /usr/bin/timelimit - effectively limit the absolute execution time of a process

 Options:

 -p      If the child process is terminated by a signal, timelimit
         propagates this condition, i.e. sends the same signal to itself. 
         This allows the program executing timelimit to determine 
         whether the child process was terminated by a signal or 
         actually exited with an exit code larger than 128.
 -q      Quiet operation - timelimit does not output diagnostic 
         messages about signals sent to the child process.
 -S killsig
         Specify the number of the signal to be sent to the 
         process killtime seconds after warntime has expired.  
         Defaults to 9 (SIGKILL).
 -s warnsig
         Specify the number of the signal to be sent to the 
         process warntime seconds after it has been started.  
         Defaults to 15 (SIGTERM).
 -T killtime
         Specify the maximum execution time of the process before 
         sending killsig after warnsig has been sent.  Defaults to 120 seconds.
 -t warntime
         Specify the maximum execution time of the process in 
         seconds before sending warnsig.  Defaults to 3600 seconds.

 On systems that support the setitimer(2) system call, the 
 warntime and killtime values may be specified in fractional 
 seconds with microsecond precision.
14

Better than --max-time are the --speed-limit and --speed-time options. In short, --speed-limit specifies the minimum average speed which you are willing to accept, and --speed-time specifies how long the transfer speed can remain below that limit before the transfer times out and is aborted.

  • 9
    I think neither is better, but in my use case --max-time is actually more appropriate because anything longer than 10 seconds would render my program useless. – Jorge Bucaran Dec 18 '15 at 4:16
  • I'm using curl as a library in a desktop application (not just calling it from the CLI) and to me your option was the best fit. My app has to be able to support long downloads, so a simple --max-time wasn't a good fit to detect "stuck downloads" (which are the case if, for instance, the user goes offline while a download is in progress), so I went with a speed-limit of 1024 and a speed-time of 30 to detect these. – André Morujão Jul 5 '17 at 10:54
  • 1
    Useful? Certainly. Better? I think that's very dependent upon your requirements – Brian Agnew Nov 10 '17 at 11:39
  • Timeouts are a problem if the response could be a large download of unknown (or even known!) size. maxtime will timeout if the large download takes eg longer than 15 minutes. And the speed-limits can be tripped by proxies that cache the entire response first before forwarding anything on. They sometimes seem to forward 1 byte on per minute, but how do you tell if that is a caching-proxy on a fast network, or a very slow connection that should be retried?? So in the end I gave up and turned off timeouts for download queries. Not sure if there is a better way. – user92979 Mar 15 '18 at 2:25
2

If you have coreutils installed on MacOS you can use the GNU timeout command which is included with that package. The GNU tools are all prefixed with a g so the CLI would be gtimeout.

gtimeout --help
Usage: gtimeout [OPTION] DURATION COMMAND [ARG]...
 or:  gtimeout [OPTION]
Start COMMAND, and kill it if still running after DURATION.

Example

$ gtimeout 1s curl -I http://www.google.com/
HTTP/1.1 200 OK
Date: Wed, 31 Oct 2018 03:36:08 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2018-10-31-03; expires=Fri, 30-Nov-2018 03:36:08 GMT; path=/; domain=.google.com
HttpOnly
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding
0

Couple of solutions in BASH4+

# -- server available to check via port xxx ?  --
function isServerAvailableNC() {
    max_secs_run="${3}"
    if timeout $max_secs_run nc -z ${1} ${2} 2>/dev/null >/dev/null; then
        #echo "${1} ✓"
        true
   else
        #echo "${1} ✗"
        return
   fi
}


# -- server available to check via port xxx ?  --
# -- supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE) --
#/usr/bin/curl -sSf --max-time 3 https://ifwewanted.to.confirm.https.com/ --insecure

function isServerAvailableCURL() {

    max_secs_run="${3}"

    proto="http://"
    if [ ! -z ${2} ] || [ ${2} -gt 80 ] ;then
        proto="https://"
    fi

    if /usr/bin/curl -sSf --max-time "${max_secs_run}" "${1}" --insecure 2>/dev/null >/dev/null; then
        #echo "${1} ✓"
        true
    else
        #echo "${1} ✗"
        false
    fi
}

Sample use:

RECOMMEND THAT NC used if we need a specific port

host="1.2.3.4"
if isServerAvailableCURL "$host" "80" "3";then
    check_remote_domain_cert "$host"
fi


host="1.2.3.4"
if isServerAvailableNC "$host" "80" "3";then
    check_remote_domain_cert "$host"
fi

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.