3

I have set up one script to write to a named pipe, then four scripts to read the named pipe.

For most of the entries, there are no problems, but when there is little processing to be done on the lines of input, some of the reads start to conflict with each other and the resulting text is a combination of random characters from (up to) 4 lines for the (up to) 4 processes.

The reason I can't use xargs or parallel, is that in the connection time to the database takes 1 second, which adds up as there are usually ~1300 lines that need to be processed.

Example Output:

Completed Runstats on MAIN.MK37 in 1s: DB20000I The RUNSTATS command completed successfully.

Completed Runstats on MAIN.MKAP in 0s: DB20000I The RUNSTATS command completed successfully.

Completed Runstats on MAIN.MKAL in 0s: DB20000I The RUNSTATS command completed successfully.

Completed Runstats on MAIN.MK49 in 0s: DB20000I The RUNSTATS command completed successfully.

Failed Runstats on AI.K1MAIN after 0s: SQL2306N The table or index "AI.K1MAIN" does not exist.

Failed Runstats on MNM5. after 0s: SQL0104N An unexpected token "MNM5." was found following "TBLE". Expected tokens may include: "". SQLSTATE=42601

Completed Runstats on MAIN.MK50 in 0s: DB20000I The RUNSTATS command completed successfully.

Completed Runstats on MAIN.MK52 in 0s: DB20000I The RUNSTATS command completed successfully.

Writer:

# Open pipe for writing
exec 10>"${PIPE_LOCATION}"

# Feed data
while read LINE; do
    echo "${LINE}" >&10
done <<< "${LIST}"

# Tell threads to stop
i=0
while [ "${i}" -lt 4 ]; do
    echo "stop" >&10
    (( i += 1 ))
done

# Close pipe
exec 10>&-

Reader:

# Open Pipe
exec 10<"${PIPE_LOCATION}"

while read -u 10 -r SCHEMA TABLE; do
    if [[ "${SCHEMA}" == 'stop' ]]; then
        break
    fi

    #
    # Runstats Code Here
    #

done

# Close Pipe
exec 10<&-

Is there some way to make the reads atomic?

I was thinking I could coordinate the main thread like a server, and the four readers could send it a request every time it wanted some input, which would probably solve the problem (5 pipes instead of 1, though!), but if there's an easier solution someone can suggest, I'm all ears!

  • Quoting from man bash: "Redirections using file descriptors greater than 9 should be used with care, as they may conflict with file descriptors the shell uses internally." – Hauke Laging Feb 10 '15 at 19:01
  • 4
    I think the only reasonable way to make this work is to write and read fixed-sized records. – Mark Plotnick Feb 10 '15 at 19:24
  • @MarkPlotnick, I could try padding them with whitespace. I'll try that, and see. – earthiverse Feb 10 '15 at 19:43
  • @HaukeLaging, thanks for the heads up! I'll try this as well. EDIT: Tried 5 instead of 10 (on both scripts), still received errors. – earthiverse Feb 10 '15 at 19:44
2

@MarkPlotnick had the right idea. I changed the script to pad the line with spaces and make read use fixed-size records.

Note: File descriptor is now 5 instead of 10, due to @Hauke Laging's findings

I now feed the data like so:

# Feed data
while read LINE; do
    printf "%64s" "${LINE}" >&5
done <<< "${LIST}"

And read it like so:

while read -u 5 -N 64 -r LINE; do
    LINE=(${LINE})
    SCHEMA=${LINE[0]}
    TABLE=${LINE[1]}

and there are no more errors!

  • 2
    The shell's read command is a performance nightmare. May not matter in your situation but in general something like awk is more suitable. – Hauke Laging Feb 10 '15 at 20:53
  • @HaukeLaging Thanks for the heads up. It's already about 50% faster and is a lot nicer on the database from it's original implementation, so I'm not too worried. I'll try to keep this info in the back of my mind if I ever need to revisit or refactor something else. – earthiverse Feb 11 '15 at 15:19
  • @HaukeLaging - while I agree about the shell, awk is pretty crazy slow compared to a real performer - such as dd. It comes with a read/write pas argument as well - with conv=sync. It's designed for this work. – mikeserv Apr 11 '15 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.