2

I was checking possibilities of pid leaks about a bash script which continuously create background jobs yet not call wait command, I happened found (by strace) that Bash monitors SIGCHLD and automatically calls wait4(...), despite that my script have not called wait command. This is why no pid leaks, that is good. But then I started think that what if I call wait command for that background pid? it does not exist in /proc, it should return error, how does Bash handle that? I did some experiments on Bash 4.4.19 and 5.1.16, turns out that Bash wait command actually gets result from background job cache, I also checked the source code, e.g., Bash 5.1.16 , see the builtins/wait.def line 253

status = wait_for_single_pid (pid, wflags|JWAIT_PERROR);

then job.c line 2611

r = bgp_search (pid);

which means

/* Search for PID in the list of saved background pids; return its status if
   found.  If not found, return -1.  We hash to the right spot in pidstat_table
   and follow the bucket chain to the end. */

.

My experiment is

test1:

bash <<'EOF'
  bash -c 'sleep 1; exit 9' &
  PID=$!
  echo $PID
  sleep 2
  ls -d /proc/$PID
  wait $PID
  echo wait result: $?
EOF

The result is:

16079
ls: cannot access '/proc/16079': No such file or directory
wait result: 9

This is an evidence that the Bash wait command use the cache (I'v also use strace to verify that, it clearly showed the last syscall of wait4 returned -1.

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 9}], 0, NULL) = 397
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 399
wait4(-1, 0x7ffd79fc3fd0, WNOHANG, NULL) = -1 ECHILD (No child processes)

Of course if I run disown -a before wait, then the wait will return code 127: wait: pid xxxxxx is not a child of this shell, this also verified that once removed the background pid from background job list then the wait command will not the exit code correctly.

So that is the conclusion that Bash wait command is using the cached result in background job management info.

Then my question become: if a script creates background jobs continuously, e.g.,

test2:

while true; do
  echo hi &
done

then the background job cache will be bigger and bigger, then will it become a memory leak?

I've tested the script, seems no memory leak, then why no leak?

EDIT: Let me make it more clear, the above script is supposed to run out of memory, but it actually does not as my observation, why?

EDIT: The above test2 is still the most interesting question, why it does not run out of memory?

EDIT: I made another test, it does run out of memory several seconds later:

test3:

bash <<'EOF'
  while true; do
    sleep 10 &
    echo $!
  done
EOF

The result is

...
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: Interrupted system call

OK, now it behaves as expected: running out of memory.

Sorry then my question becomes: is this by design? I have never heard of the caveat of continuously create background job. The only solution I know so far is use disown to stop managing the background jobs, or use other tricks such as (cmd&) to start a process without being managed as background job.

EDIT: answered by myself: this should be by design, it just means Bash tracking all active jobs, if in a short period, there are a lot of active jobs, then it runs out of memory. So this is not contradictory to the test2.

EDIT: added another test to show that the Bash background job exit code cache does not just cache the exit code of last job besides of active jobs, it do cache exit code of all jobs.

test4:

bash -x <<'EOF'
  bash -c '/bin/sleep 3; exit 1' &
  PID1=$!
  bash -c '/bin/sleep 6; exit 2' &
  PID2=$!
  wait $PID1
  echo exit code of first process is: $?
  wait $PID2
  echo exit code of second process is: $?
  wait $PID1
  echo Get exit code of first process again, result is: $?
EOF

result is:

+ PID1=2357449
+ bash -c '/bin/sleep 3; exit 1'
+ PID2=2357450
+ wait 2357449
+ bash -c '/bin/sleep 6; exit 2'
+ echo exit code of first process is: 1
exit code of first process is: 1
+ wait 2357450
+ echo exit code of second process is: 2
exit code of second process is: 2
+ wait 2357449
+ echo Get exit code of first process again, result is: 1
Get exit code of first process again, result is: 1
5
  • You should show in your question that if you run wait $PID1 AGAIN at the end, after you waited for PID2, it would show the exit status of PID1, which proves that bash keeps the results of the the children in the memory, and that's your main point.
    – aviro
    Commented May 7, 2023 at 15:38
  • @aviro thanks for taking a look at it. I have updated the test code and result as your suggestion.
    – osexp2000
    Commented May 7, 2023 at 15:45
  • I still don't understand is that why test2 does not run out of memory? based on my observation, Bash caches all exit codes. There might be something I still don't know.
    – osexp2000
    Commented May 7, 2023 at 15:51
  • 1
    I hope it's clear now that test3 didn't "run out of memory", it just hit the nproc limit. So it wasn't related to bash or to your question at all.
    – aviro
    Commented May 8, 2023 at 7:14
  • @aviro yes, I have updated EDIT part of the question, test3 is not contradictory with test2. The reason why test2 does not run out of memory maybe just because it takes much time or Bash did some cleanup work in some timing.
    – osexp2000
    Commented May 8, 2023 at 10:27

2 Answers 2

2

First, the memory will never run out because the maximum pid (which you can see in /proc/sys/kernel/pid_max) is limited, usually 32768. So even if you run more processes then that, eventually the pids will get recycled by the kernel, so that largest number of pids your bash would keep in the memory would be < 32768.

In bash the size also depends on your nproc (The maximum user processes) limit.

And you can easily confirm it's true in your bash using the following script:

#!/bin/bash
declare -a pids_list=()
for i in {1..4196}; do
    (exit 0) & waitpid=$! && wait $waitpid
    pids_list+=($waitpid)
done
export KEPT=0 DISCARDED=0
for i in "${pids_list[@]}"
do
    wait $i 2>/dev/null
    if [ $? -ne 127 ] # If the child is not found in the jobs table, wait returns 127
    then
        let KEPT++
    else
        let DISCARDED++
        KEPT=0
    fi
done
echo KEPT=$KEPT DISCARDED=$DISCARDED

In this case, I run 4096+100=4196 jobs in the background and wait for each one to finish, while keeping the pids in the pids_list array. After all jobs finished, I loop over the pids_list array and check if bash still keeps it's status.

In my case, my default max proc limit is 4096:

$ ulimit -u
4096

If I run this code (either as a script, or if you source it), it will confirm that it would discard the status of the first 100 pids, and only keep the last 4096 pids in the memory:

$ check_pid_table.sh
KEPT=4096 DISCARDED=100

If I decrease the limit to 1024, that's the number of processes it would keep.

$ ulimit -u 1024
$ check_pid_table.sh
KEPT=1024 DISCARDED=3172

And if I increase the limit, it would keep all pids (but again - up to your pid_max limit).

$ ulimit -u 8192
$ check_pid_table.sh
KEPT=4196 DISCARDED=0

How much memory does the process table take in bash

You can also check how much memory is being consumed by bash for different number of pids it needs to keep. I'm using here the time(1) command to check the memory used by the bash process.

%M in time checks the Maximum resident set size of the process during its lifetime, in Kbytes.

$ ulimit -u 65536
$ for i in {1..8} {25..32}; do 
>   /usr/bin/time -f "number of procs=$i KB, memory=%M KB" bash -c '
>     for (( i=$0 ; i>0 ; i-- )); do 
>       echo >/dev/null & wait $!
>     done' $(($i*1024))
> done
number of procs=1 KB, memory=2904 KB
number of procs=2 KB, memory=2936 KB
number of procs=3 KB, memory=2968 KB
number of procs=4 KB, memory=3000 KB
number of procs=5 KB, memory=3032 KB
number of procs=6 KB, memory=3064 KB
number of procs=7 KB, memory=3096 KB
number of procs=8 KB, memory=3128 KB
number of procs=25 KB, memory=3672 KB
number of procs=26 KB, memory=3704 KB
number of procs=27 KB, memory=3736 KB
number of procs=28 KB, memory=3768 KB
number of procs=29 KB, memory=3796 KB
number of procs=30 KB, memory=3796 KB
number of procs=31 KB, memory=3796 KB
number of procs=32 KB, memory=3796 KB

You can see that each chunk of 1K processes add around 32K to the memory consumption of bash, which means that every process entry uses 32 bits.

However, when we're getting close to 32 KB (which is the max_pid limit), you can see the memory becomes static. That's because, as I said, eventually the pids get recycled (and there are already many processes running on my system).

5
  • Cygwin (GNU bash, version 4.4.12(3)-release): KEPT=512 DISCARDED=3684. Raspbian (GNU bash, version 5.1.4(1)-release): KEPT=4196 DISCARDED=0 Commented May 9, 2023 at 11:53
  • @roima, I guess it differs on the bash version. In your second version, try to increase the number of pids to see if the limit is different. Also, in the version maybe wait returns a different code if the pid is not in the table, try to check exit code different than zero and not 127.
    – aviro
    Commented May 9, 2023 at 12:01
  • @roima, my bad; The 4096 limit wasn't because of bash; It was because of the maxproc limit
    – aviro
    Commented May 9, 2023 at 13:26
  • @aviro Excellent! Thanks so much for you investigation. That explains all clearly. I have reproduced exact same results by your test code.
    – osexp2000
    Commented May 10, 2023 at 9:27
  • This behavior of Bash is convenient, but a little bit creepy, say if you happened spawned 2 background processes, they happened get same pid, then at last the wait $PID will get exit code of the last process, not the first one, sounds good for almost all users, I am fine, but just need clarification, it is not that obvious to get relieved. Thanks for all of you, I am completely relieved now.
    – osexp2000
    Commented May 10, 2023 at 9:33
2

It's not a leak, no, because the memory isn't lost. The shell tracks the PIDs so in theory it could eventually run out of memory, but all of it is expected and managed memory use.

Under POSIX the shell only tracks active PIDs and the result of the last PID to exit:

wait [-n] [n ...] Wait for each specified child process and return its termination status. Each n may be a process ID or a job specification; if a job spec is given, all processes in that job's pipeline are waited for. If n is not given, all currently active child processes are waited for, and the return status is zero. If the -n option is supplied, wait waits for any job to terminate and returns its exit status. If n specifies a non-existent process or job, the return status is 127. Otherwise, the return status is the exit status of the last process or job waited for.

In the case of bash (at least 4.4.12 through 5.2) though, you are correct in that it does not follow POSIX, and even with bash --posix the behaviour is not POSIX compliant. Instead, all background process statuses are retained. This is successfully demonstrated in your "test4". Contrast the result by using POSIX-compliant dash:

exit code of first process is: 1
exit code of second process is: 2
Get exit code of first process again, result is: 127

Looking at the source code for bash, in the file nojobs.c and in particular the function alloc_pid_list (called from wait_builtin in wait.def), each managed PID uses an additional 12 byte entry in the array pid_list. You are highly unlikely to run your system out of resources due to this array growing in size before you run it out for other reasons.

4
  • Maybe Bash has a cache max limit of exited process?
    – osexp2000
    Commented May 7, 2023 at 15:53
  • @osexp2003 today I have discovered that bash has another part that is not POSIX-compliant. (Actually, the documentation is at odds with the reality too, so something's not right.) Commented May 7, 2023 at 17:12
  • Thank you very much for investigation. Good to know that the POSIX-compliant dash works as expected, I am mostly relieved. But the reason why test2 did not run out of memory is probably Bash did some cleanup work we are not aware, instead of the additional 12 byte entry in the array pid_list is very small. If not getting cleaned up, even just 1 additional byte in pid_list could cause memory increasing,
    – osexp2000
    Commented May 8, 2023 at 3:11
  • Actually in the test2, I watched the output of top, it showed that test2 memory did not increase at all, it forked 226981 background processes, each pid was unique, no chance for cleanup on duplicates. I believe that Bash did some cleanup on other timing, such as there is reap_dead_jobs function, it may be called in some conditions. It is hard to fully analyze the source code, and Bash handled the dead jobs well, I am going to close this question
    – osexp2000
    Commented May 8, 2023 at 3:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .