0

I've been greatly inspired by this question: Parallelize a Bash FOR Loop to parallelise some tools I've written that involve very loooong while read loops (ie doing the same task / set of tasks across paths given in an input file. The input file would contain around 90,000 lines, and growing).

I've done all the work to 'shoehorn' PSkocik's 'N processes with a FIFO-based semaphore' example into my code...

# initialize a semaphore with a given number of tokens
open_sem(){
    mkfifo /tmp/pipe-$$
    exec 3<>/tmp/pipe-$$
    rm /tmp/pipe-$$
    local i=$1
    for((;i>0;i--)); do
        printf %s 000 >&3
    done
}

# run the given command asynchronously and pop/push tokens
run_with_lock(){
    local x
    # this read waits until there is something to read
    read -u 3 -n 3 x && ((0==x)) || exit $x
    (
     ( "$@"; )
    # push the return code of the command to the semaphore
    printf '%.3d' $? >&3
    )&
}

N=4
open_sem $N
for thing in {a..g}; do
    run_with_lock task $thing
done 

However, my 'old' code had a nice progress counter built into the read loop (code below is abbreviated) and yes, I know there is a weird mix of echo, awk and printf I tend to reuse code from other scripts I've written where maybe some of the code was based off of other online examples etc... I'm sure I can tidy this up... but its works and I'm the only one using this code!:

## $temp1 is the file with 90,000 lines to read over
## $YELLOW is a global variable exported from my bashrc with the escape code for yellow text
## $GREEN is a global variable exported from my bashrc with the escape code for green text
## $CL is a global variable exported from my bashrc with the escape code for Clear Line
## $NC is a global variable exported from my bashrc with the escape code to revert text colour back to normal

num_lines="$(cat $temp1 | wc -l)"
percent_per_line="$(awk "BEGIN {print 100/$num_lines}")"
progress_percent='0'
current_line='1'

echo -ne "${CL}${YELLOW}PROGRESS: ${progress_percent}% ${NC}\r"
while read line; do
    ############################################
    ##commands to process $line data as needed##
    ############################################

    progress_percent="$(awk "BEGIN {print $progress_percent + $percent_per_line }")"
    awk -v y=$YELLOW -v nc=$NC -v progress=$progress_percent -v current_line=$current_line -v total_lines=$num_lines 'BEGIN {printf (y"\033[2KPROGRESS: %.3f%%   (%d OF %d)\n\033[1A"nc, progress, current_line, total_lines) }' 
    #I think I mixed my global var escape codes with typed out ones cause I was I forgot / have no need to export \033[2K and \033[1A for anything else?
    ((current_line++))
done < "$temp1"

echo -e "${CL}${GREEN}PROGRESS: 100.000%   (${num_lines} OF ${num_lines})${NC}"

I'm trying to find a way to again, shoehorn something with a similar output into the 'new' FIFO-semaphore code....

I just can't work out how! does it go into the run_with_lock function, if so where, and I would need to pass into that function the percent_per_line and num_lines variables but it passes $@ within it :( I feel I'm not fully understanding how the FIFO-semaphore works as if I did I would use another semaphore message type thing to pass the data I need around...

Any help would be massively appreciated as it helps me learn and improve!!

1 Answer 1

0

Ok, so I feel dumb... Whilst I'm sure its not 100% accurate, after doing a lot of testing with a simply test script I feel I've got to grips a little with this.

Something I wasn't fully 'aware of' was the exact method of function of the whole semaphore thing. The way I saw it was that the while read loop would do all its 'reading' immediatly, adding all lines to some queue (I thought the Pipe) so my initial thought was that any other contents in the read loop would also run 'completely' (ie immediately jump to 100%) and the script would then move to the wait after the read loop for the queue to be finished, leaving me with a terminal screen saying 100% done, even though its still actually doing the work.

As its turns out, I was very wrong! the run_with_lock means the while read loop only reads N lines at a time, so any code after the run_with_lock call only runs intermittently as and when the lock is released.

As such, to achieve my goal all I need to do is keep my progress code in the exact same place and all is ok.... no special faffing that I thought was required.

The drawback is the progress now shows when items start being processed, not updating AFTER they have actually finished, but that's tiny with respect to what's needed

You must log in to answer this question.