In the C API, system calls return a negative value to indicate an error, and the error code in errno
gives more information on the nature of the error. Your man page should explain the possible errors on your system. There are two standard error codes:
EAGAIN
indicates that the new process cannot be created due to a lack of available resources, either insufficient memory of some kind or a limit that has been reached such as the maximum number of processes per user or overall.
ENOMEM
indicates that the new process cannot be created due to a lack of memory of some kind. Under Linux, ENOMEM
indicates a lack of kernel memory, while a lack of userland memory is reported as EAGAIN
. Under OpenBSD, ENOMEM
is used for a lack of userland memory as well.
In summary, fork
can fail due to a lack of available resources (possibly in the form of an artificial limit rather than a simple lack of memory).
The behavior of shells when fork
fails is not specified by POSIX. In practice, they tend to return different error statuses (1 in pdksh, 2 in ash, 254 in bash, 258 in ksh93, 1 in tcsh; zsh returns 0 which is a bug). If you're implementing a shell for production use (as opposed to a learning exercise), it might be worth discussing this on the Austin Group mailing list.