1

I have a FIFO question:

Why the fifo thread_num limit 2, but number of processes, sustained growth when one process ends, two processes start, and then got 3 processes.

sustained growth 2n-1

#!bin/bash
#exec clear file
mkdir -p xxx
thread_num=2
 
[ ! -p tmp ] && mkfifo tmp
exec 9<>tmp
 
for ((i=0;i<$thread_num;i++)); do
    echo >&9
done
echo "================start sh===================="
for i in  `cat shfiles.txt`;
do
        read -u 9
        {
                sh $i
                mv $i xxx/
                echo >&9
        } &
done
wait
exec 9>&-
rm tmp
echo "=================done===================="
exit 0

@Paul_Pedant
I can't add comment now. This code is simplified, and I execute it in a folder, using the root account.

shfiles.txt is a collection of SH file paths.

In the executed file is the perforce command, like this:

20210326/archive13.sh

20210326/archive14.sh

p4 archive -D depot -t xxxxx#1 

I use ps -aux | grep sh to view the current process, which is growing and ending.

Two processes are generated at the end of each process.  So there are more and more processes.

I appear to have solved it, but I don't know why. To modify the fifo 9 -> 7 like exec 7<>tmp. Is that 9 Used by other processes?

Thank you all for helping me. I got ps like this

sh -x ShArchiveCommand.sh 
sh -x ShArchiveCommand.sh 
sh -x ShArchiveCommand.sh 
20210326/ArchiveTList12.sh
20210326/ArchiveTList13.sh

and then after one hour

sh -x ShArchiveCommand.sh 
sh -x ShArchiveCommand.sh 
20210326/ArchiveTList12.sh
sh -x ShArchiveCommand.sh 
20210326/ArchiveTList14.sh
sh -x ShArchiveCommand.sh
20210326/ArchiveTList15.sh

and I check ArchiveTList12 not finished yet

a new exception

sh -x ShArchiveCommand.sh 
sh -x ShArchiveCommand.sh 
20210326/ArchiveTList12.sh
sh -x ShArchiveCommand.sh 
20210326/ArchiveTList13.sh
sh -x ShArchiveCommand.sh 
20210326/ArchiveTList14.sh

and I check ArchiveTList12,ArchiveTList13 not finished all

6
  • 1
    You may want to run that script under env SHELLOPTS=xtrace ./that-script to see what's going on. Mar 29, 2021 at 11:09
  • [A] What are the contents of shfiles.txt? Each line is being run by sh, and could be doing anything they please. [B] How are you counting processes? ps and pstree will tell you actual running processes. But Bash (your outer script) will aggressively harvest child processes (so they disappear from ps) but continue to list them in the jobs list until the wait is invoked, which is after all the input has been read. Mar 29, 2021 at 14:57
  • 1
    This has many issues. [A] The shebang is relative, and only valid when run in root directory, in which case the mkdir won't work. Maybe you are testing raw scripts as root in / ? [B] It throws six errors in shellcheck. [C] The cat will pass each input word (not line) to dash separately. Mar 29, 2021 at 15:33
  • Points on your edits to the original post. [1] You are running the scripts under sh, which is usually a link to dash, ksh, or bash depending on your distro. So I don't know how to replicate your problem exactly. [2] Your wait is not called until after the for .. done loop completes. So probably the shells you see in ps are finished, but awaiting having their status requested. (Bash avoids this by collecting and storing results so ps does not see them.) [3] You get two shells per iteration -- a subshell running the { ... } & and the one running archive??.sh. Mar 30, 2021 at 19:24
  • 2
    @wangjiaying you appear to have two accounts 1 and 2. Please complete the registration process for one of them, then Contact SE to merge them so that you can continue managing your question. Thank you!
    – Jeff Schaller
    Mar 31, 2021 at 15:04

1 Answer 1

0

Interesting methodology, so I followed it up.

I have a diagnostic script that illustrates the operation. In particular, it labels each Tick ("permit to run") with a sequence number and time, like [4] Sent 19:53:54, so when it is read, it can be identified. As the sequence ends when the shfiles.txt ends, I also flush remaining Ticks.

I also marked all debug with the prefix [ x ] && . These lines can be permanently removed, or just disabled by changing the x to a space. And I changed all the filenames.

My commands file moonie.cmds is this (each command identifies itself and makes the scheduling ragged):

echo One && sleep 8
echo Two && sleep 3
echo Three && sleep 6
echo Four && sleep 5
echo Five && sleep 9

These are simple commands, which I run using bash -c. Script names should be OK, but args and quoting in the .cmds file may be difficult. I couldn't see why you mv the scripts (?) to another directory after they are run, as I don't know what your .cmds file looks like.

I used pstree to show that my bash and outer script pids are constant, and the asynchronous subshell and the sleep run two at a time, as configured by the threads variable. I tested with 1, 2, 3, and 10 threads.

My script is this:

#! /bin/bash

Threads=2

mkdir -p moonie.dir     #.. Not used.

#.. Clean the named pipe and attach to fd9. 
rm -f moonie.fifo
mkfifo moonie.fifo
exec 9<>moonie.fifo
 
#.. Seed the fifo with required thread count.
for ((Seq = 1; Seq <= Threads; ++Seq)); do
    printf '[%s] Sent %(%T)T\n' "${Seq}" -1 >&9
done

echo "================ start sh ===================="

[ x ] && sleep 1    #.. Show we are waiting for the Ticks.

while IFS='' read -r Cmd; do    #.. Work through the command list.

    IFS='' read -r -u 9 Tick    #.. Wait for a free thread. 
    [ x ] && printf '%(%T)T Read Tick %s\n' -1 "${Tick}"

    {   #.. Launch a thread asynchronously.
        [ x ] && printf '%(%T)T Begin %s\n' -1 "${Cmd}"
        bash -c "${Cmd}"
        [ x ] && printf '%(%T)T Ended %s\n' -1 "${Cmd}"
        #.. Refresh its thread on completion.
        printf '[%s] Sent %(%T)T\n' "${Seq}" -1 >&9
    } &
    (( ++Seq ))     #.. Number Ticks for diagnostics.

    [ x ] && sleep 0.3              #.. Wait for the job to register.
    [ x ] && pstree -p "${PPID}"    #.. Show the running job hierarchy.

done < moonie.cmds

wait    #.. Wait for all threads to complete.

#.. Flush superfluous Ticks (timeout read because the fifo will hang).
while IFS='' read -r -t 1 -u 9 Tick; do
    [ x ] && printf '%(%T)T Discard Tick %s\n' -1 "${Tick}"
done

exec 9>&-       #.. Disconnect and remove the fifo.
rm moonie.fifo

echo "================= done ===================="
exit 0

The debug session was this:

================ start sh ====================
19:53:51 Read Tick [1] Sent 19:53:50
19:53:51 Begin echo One && sleep 8
One
bash(12037)───moonie(16150)─┬─moonie(16155)───sleep(16157)
                            └─pstree(16158)
19:53:51 Read Tick [2] Sent 19:53:50
19:53:51 Begin echo Two && sleep 3
Two
bash(12037)───moonie(16150)─┬─moonie(16155)───sleep(16157)
                            ├─moonie(16159)───sleep(16161)
                            └─pstree(16162)
19:53:54 Ended echo Two && sleep 3
19:53:54 Read Tick [4] Sent 19:53:54
19:53:54 Begin echo Three && sleep 6
Three
bash(12037)───moonie(16150)─┬─moonie(16155)───sleep(16157)
                            ├─moonie(16164)───sleep(16166)
                            └─pstree(16167)
19:53:59 Ended echo One && sleep 8
19:53:59 Read Tick [3] Sent 19:53:59
19:53:59 Begin echo Four && sleep 5
Four
bash(12037)───moonie(16150)─┬─moonie(16164)───sleep(16166)
                            ├─moonie(16168)───sleep(16170)
                            └─pstree(16171)
19:54:00 Ended echo Three && sleep 6
19:54:00 Read Tick [5] Sent 19:54:00
19:54:00 Begin echo Five && sleep 9
Five
bash(12037)───moonie(16150)─┬─moonie(16168)───sleep(16170)
                            ├─moonie(16172)───sleep(16174)
                            └─pstree(16175)
19:54:04 Ended echo Four && sleep 5
19:54:09 Ended echo Five && sleep 9
19:54:09 Discard Tick [6] Sent 19:54:04
19:54:09 Discard Tick [7] Sent 19:54:09
================= done ====================

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .