Bash wait command actually gets result from background job cache, would the cache cause memory leak when it creates background jobs continuously?

Question

I was checking possibilities of pid leaks about a bash script which continuously create background jobs yet not call wait command, I happened found (by strace) that Bash monitors SIGCHLD and automatically calls wait4(...), despite that my script have not called wait command. This is why no pid leaks, that is good. But then I started think that what if I call wait command for that background pid? it does not exist in /proc, it should return error, how does Bash handle that? I did some experiments on Bash 4.4.19 and 5.1.16, turns out that Bash wait command actually gets result from background job cache, I also checked the source code, e.g., Bash 5.1.16 , see the builtins/wait.def line 253

status = wait_for_single_pid (pid, wflags|JWAIT_PERROR);

then job.c line 2611

r = bgp_search (pid);

which means

/* Search for PID in the list of saved background pids; return its status if
   found.  If not found, return -1.  We hash to the right spot in pidstat_table
   and follow the bucket chain to the end. */

.

My experiment is

test1:

bash <<'EOF'
  bash -c 'sleep 1; exit 9' &
  PID=$!
  echo $PID
  sleep 2
  ls -d /proc/$PID
  wait $PID
  echo wait result: $?
EOF

The result is:

16079
ls: cannot access '/proc/16079': No such file or directory
wait result: 9

This is an evidence that the Bash wait command use the cache (I'v also use strace to verify that, it clearly showed the last syscall of wait4 returned -1.

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 9}], 0, NULL) = 397
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 399
wait4(-1, 0x7ffd79fc3fd0, WNOHANG, NULL) = -1 ECHILD (No child processes)

Of course if I run disown -a before wait, then the wait will return code 127: wait: pid xxxxxx is not a child of this shell, this also verified that once removed the background pid from background job list then the wait command will not the exit code correctly.

So that is the conclusion that Bash wait command is using the cached result in background job management info.

Then my question become: if a script creates background jobs continuously, e.g.,

test2:

while true; do
  echo hi &
done

then the background job cache will be bigger and bigger, then will it become a memory leak?

I've tested the script, seems no memory leak, then why no leak?

EDIT: Let me make it more clear, the above script is supposed to run out of memory, but it actually does not as my observation, why?

EDIT: The above test2 is still the most interesting question, why it does not run out of memory?

EDIT: I made another test, it does run out of memory several seconds later:

test3:

bash <<'EOF'
  while true; do
    sleep 10 &
    echo $!
  done
EOF

The result is

...
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: Interrupted system call

OK, now it behaves as expected: running out of memory.

Sorry then my question becomes: is this by design? I have never heard of the caveat of continuously create background job. The only solution I know so far is use disown to stop managing the background jobs, or use other tricks such as (cmd&) to start a process without being managed as background job.

EDIT: answered by myself: this should be by design, it just means Bash tracking all active jobs, if in a short period, there are a lot of active jobs, then it runs out of memory. So this is not contradictory to the test2.

EDIT: added another test to show that the Bash background job exit code cache does not just cache the exit code of last job besides of active jobs, it do cache exit code of all jobs.

test4:

bash -x <<'EOF'
  bash -c '/bin/sleep 3; exit 1' &
  PID1=$!
  bash -c '/bin/sleep 6; exit 2' &
  PID2=$!
  wait $PID1
  echo exit code of first process is: $?
  wait $PID2
  echo exit code of second process is: $?
  wait $PID1
  echo Get exit code of first process again, result is: $?
EOF

result is:

+ PID1=2357449
+ bash -c '/bin/sleep 3; exit 1'
+ PID2=2357450
+ wait 2357449
+ bash -c '/bin/sleep 6; exit 2'
+ echo exit code of first process is: 1
exit code of first process is: 1
+ wait 2357450
+ echo exit code of second process is: 2
exit code of second process is: 2
+ wait 2357449
+ echo Get exit code of first process again, result is: 1
Get exit code of first process again, result is: 1

Answer 1

First, the memory will never run out because the maximum pid (which you can see in /proc/sys/kernel/pid_max) is limited, usually 32768. So even if you run more processes then that, eventually the pids will get recycled by the kernel, so that largest number of pids your bash would keep in the memory would be < 32768.

In bash the size also depends on your nproc (The maximum user processes) limit.

And you can easily confirm it's true in your bash using the following script:

#!/bin/bash
declare -a pids_list=()
for i in {1..4196}; do
    (exit 0) & waitpid=$! && wait $waitpid
    pids_list+=($waitpid)
done
export KEPT=0 DISCARDED=0
for i in "${pids_list[@]}"
do
    wait $i 2>/dev/null
    if [ $? -ne 127 ] # If the child is not found in the jobs table, wait returns 127
    then
        let KEPT++
    else
        let DISCARDED++
        KEPT=0
    fi
done
echo KEPT=$KEPT DISCARDED=$DISCARDED

In this case, I run 4096+100=4196 jobs in the background and wait for each one to finish, while keeping the pids in the pids_list array. After all jobs finished, I loop over the pids_list array and check if bash still keeps it's status.

In my case, my default max proc limit is 4096:

$ ulimit -u
4096

If I run this code (either as a script, or if you source it), it will confirm that it would discard the status of the first 100 pids, and only keep the last 4096 pids in the memory:

$ check_pid_table.sh
KEPT=4096 DISCARDED=100

If I decrease the limit to 1024, that's the number of processes it would keep.

$ ulimit -u 1024
$ check_pid_table.sh
KEPT=1024 DISCARDED=3172

And if I increase the limit, it would keep all pids (but again - up to your pid_max limit).

$ ulimit -u 8192
$ check_pid_table.sh
KEPT=4196 DISCARDED=0

How much memory does the process table take in bash

You can also check how much memory is being consumed by bash for different number of pids it needs to keep. I'm using here the time(1) command to check the memory used by the bash process.

%M in time checks the Maximum resident set size of the process during its lifetime, in Kbytes.

$ ulimit -u 65536
$ for i in {1..8} {25..32}; do 
>   /usr/bin/time -f "number of procs=$i KB, memory=%M KB" bash -c '
>     for (( i=$0 ; i>0 ; i-- )); do 
>       echo >/dev/null & wait $!
>     done' $(($i*1024))
> done
number of procs=1 KB, memory=2904 KB
number of procs=2 KB, memory=2936 KB
number of procs=3 KB, memory=2968 KB
number of procs=4 KB, memory=3000 KB
number of procs=5 KB, memory=3032 KB
number of procs=6 KB, memory=3064 KB
number of procs=7 KB, memory=3096 KB
number of procs=8 KB, memory=3128 KB
number of procs=25 KB, memory=3672 KB
number of procs=26 KB, memory=3704 KB
number of procs=27 KB, memory=3736 KB
number of procs=28 KB, memory=3768 KB
number of procs=29 KB, memory=3796 KB
number of procs=30 KB, memory=3796 KB
number of procs=31 KB, memory=3796 KB
number of procs=32 KB, memory=3796 KB

You can see that each chunk of 1K processes add around 32K to the memory consumption of bash, which means that every process entry uses 32 bits.

However, when we're getting close to 32 KB (which is the max_pid limit), you can see the memory becomes static. That's because, as I said, eventually the pids get recycled (and there are already many processes running on my system).

Answer 2

It's not a leak, no, because the memory isn't lost. The shell tracks the PIDs so in theory it could eventually run out of memory, but all of it is expected and managed memory use.

Under POSIX the shell only tracks active PIDs and the result of the last PID to exit:

wait [-n] [n ...] Wait for each specified child process and return its termination status. Each n may be a process ID or a job specification; if a job spec is given, all processes in that job's pipeline are waited for. If n is not given, all currently active child processes are waited for, and the return status is zero. If the -n option is supplied, wait waits for any job to terminate and returns its exit status. If n specifies a non-existent process or job, the return status is 127. Otherwise, the return status is the exit status of the last process or job waited for.

In the case of bash (at least 4.4.12 through 5.2) though, you are correct in that it does not follow POSIX, and even with bash --posix the behaviour is not POSIX compliant. Instead, all background process statuses are retained. This is successfully demonstrated in your "test4". Contrast the result by using POSIX-compliant dash:

exit code of first process is: 1
exit code of second process is: 2
Get exit code of first process again, result is: 127

Looking at the source code for bash, in the file nojobs.c and in particular the function alloc_pid_list (called from wait_builtin in wait.def), each managed PID uses an additional 12 byte entry in the array pid_list. You are highly unlikely to run your system out of resources due to this array growing in size before you run it out for other reasons.