I am stuck with some problems with my scripts in ksh. FWIW the problem which I am unable to overcome is that when I use a structure such as this
command | while read VAR1 do many.commands using $VAR1 done
I often get that my scripts do not perform the loop for every line piped to the while. To test this I then change the structure to
command > /tmp/tempfile cat -n /tmp/tempfile >&2 cat /tmp/tempfile | while read VAR1 etc
This proves that there is many lines in the output.
Further I then add an extra line immediately after the do, like
echo DEBUGGING: $VAR1 >&2
Which proves that the loop runs only once. I am really stumped.
One workaround which is not always viable, is to do
for X in $(cat /tmp/tempfile ) do ... done
This then works correctly but besides the fact that I hate this for structure, It means you expand the entire input data on the command-line (which has hard limits)
It appears that bash is better than ksh at handling this kind of thing. In particular it seems that this may be related to read calls failing but not retrying if the loop takes a long time to run.
However it seems that bash does not have a built-in "read", which means much of my scripts will need to be re-written. I OFTEN use large structures like
command1 | command2 | while read SOMEVAR; do awk -F: "... long awk program" | sed "long sed program" ; done | sort -u | tail -1 | read FINAL_ANSWER
The problem is that bash uses /usr/bin/read which as expected throws away the result of FINAL_ANSWER as fast as it gets it. The obvious workaround is to replace
| read FINAL_ANSWER
> /tmp/final_answer && FINAL_ANSWER="$(cat /tmp/final_answer)"
So.... Any scripting gurus on here able to shed some more light on this? I deliberately did not post my real scripts here both because they are part of a sensitive solution developed for a customer, and because I don't want the actual detail of the scripts to confuse the issue.
I use the "while read" format OFTEN. It usually works. I've in fact never had a problem with it in 25 years of shell scripting. Now I'm having problems. Very frustrating. Perplexing.
Initially I thought the while read is only receiving, or passing along, the first line of input. But then I discovered a situation where, when I run the script over and over, it runs further and further into the input. Specifically I have something line this
command | while read NEXT_ONE DONEFLAG do if [ $DONEFLAG = "yes" ] then echo Already completed work for $NEXT_ONE else dowork $NEXT_ONE && set_flag $NEXT_ONE fi done
It turns out that on each run of the script, it performs
dowork once. It doesn't really matter what
dowork is, so long as it takes more than a few seconds. Some kind of shell pipe timeout occurs and the rest of the input then disappears. Google tells me that dtksh may solve this issue (Apparently it will retry the read/write or something, I didn't read enough)
I see that dtksh exists in /usr/st/bin/dtksh
Who is this? I don't like using shells that I don't know, but it might be worth while to split small portions of scripts into sub-scripts with /usr/dt/bin/dtksh as interpreter.
EDIT: Providing an example of why I can not use bash as a drop-in replacement for ksh as interpreter:
sol10-primary> # cat test.sh #!/bin/ksh echo hello| read VAR1 echo $VAR1 sol10-primary> # ./test.sh hello sol10-primary> # sed 's/ksh/bash/' <test.sh >test2.sh sol10-primary> # chmod +x test2.sh sol10-primary> # ./test2.sh sol10-primary> #