I have set up one script to write to a named pipe, then four scripts to read the named pipe.
For most of the entries, there are no problems, but when there is little processing to be done on the lines of input, some of the reads start to conflict with each other and the resulting text is a combination of random characters from (up to) 4 lines for the (up to) 4 processes.
The reason I can't use xargs or parallel, is that in the connection time to the database takes 1 second, which adds up as there are usually ~1300 lines that need to be processed.
Example Output:
Completed Runstats on MAIN.MK37 in 1s: DB20000I The RUNSTATS command completed successfully.
Completed Runstats on MAIN.MKAP in 0s: DB20000I The RUNSTATS command completed successfully.
Completed Runstats on MAIN.MKAL in 0s: DB20000I The RUNSTATS command completed successfully.
Completed Runstats on MAIN.MK49 in 0s: DB20000I The RUNSTATS command completed successfully.
Failed Runstats on AI.K1MAIN after 0s: SQL2306N The table or index "AI.K1MAIN" does not exist.
Failed Runstats on MNM5. after 0s: SQL0104N An unexpected token "MNM5." was found following "TBLE". Expected tokens may include: "". SQLSTATE=42601
Completed Runstats on MAIN.MK50 in 0s: DB20000I The RUNSTATS command completed successfully.
Completed Runstats on MAIN.MK52 in 0s: DB20000I The RUNSTATS command completed successfully.
Writer:
# Open pipe for writing
exec 10>"${PIPE_LOCATION}"
# Feed data
while read LINE; do
echo "${LINE}" >&10
done <<< "${LIST}"
# Tell threads to stop
i=0
while [ "${i}" -lt 4 ]; do
echo "stop" >&10
(( i += 1 ))
done
# Close pipe
exec 10>&-
Reader:
# Open Pipe
exec 10<"${PIPE_LOCATION}"
while read -u 10 -r SCHEMA TABLE; do
if [[ "${SCHEMA}" == 'stop' ]]; then
break
fi
#
# Runstats Code Here
#
done
# Close Pipe
exec 10<&-
Is there some way to make the reads atomic?
I was thinking I could coordinate the main thread like a server, and the four readers could send it a request every time it wanted some input, which would probably solve the problem (5 pipes instead of 1, though!), but if there's an easier solution someone can suggest, I'm all ears!
man bash
: "Redirections using file descriptors greater than 9 should be used with care, as they may conflict with file descriptors the shell uses internally."