0

I have an issue with the script below. It uses a few sql statements in a for loop to set variables and produces a log file. The current db produces +- 1200 lines of output.

The original version is with the commented echo line, it adds one line to a text file (report.csv) for each loop (one write operation to disk).

The new intended version would append the data to an array (dcimlog), and then do only one write operation for the whole dataset in newreport.csv.

My issue is that somehow the array is not appended correctly by the child processes, so newreport.csv only contains the first line done before the for loop, and is not appended by any child.

The order of the lines is not important, but the speed of execution is, hence the "do () &" -- which I believe might be the reason for the failure...

Any idea how I can get this to work?

N=$(nproc)
dcimlog=();dcimlog=("${dcimlog[@]}" "some stuff") # works
for i in "${sqlanswer[@]}"; do
    ( echo somework #get the data from sql
    #   echo "$DDeviceID;$NBPort;$PortNumber;$CConnectedDevice;$Connectedport;$CabinetLocation;$HeightDevice;$PositionDevice;$ZoneName;$DataCenterName" >> report.csv
    dcimlog=("${dcimlog[@]}" "$DDeviceID;$NBPort;$PortNumber;$CConnectedDevice;$Connectedport;$CabinetLocation;$HeightDevice;$PositionDevice;$ZoneName;$DataCenterName") # does not work
) &
    if [[ $(jobs -r -p | wc -l) -gt $N ]]; then
            wait -n
    fi
done
wait
echo "${dcimlog[@]}" > newreport.csv
  • 1
    Not sure I understand correctly, but variable / array assignments in a subshell won't be passed back to the parent shell. The echo redirected to the report file should work from the subshell, so why not leave it in? – RudiC Oct 20 '18 at 13:40
  • Because with the echo in the loop, I open the file for 1200 consecutives writes and close, instead of one, if I managed to get to echo everything at once – user1747036 Oct 20 '18 at 14:33
  • 3
    Why don't you echo, and redirect the entire loop's stdout to the file? Like for FN in *; do (echo $BASHPID, $$, $FN) & done >result – RudiC Oct 20 '18 at 14:54
  • Thank you @RudiC , that worked like a charm! If you create an "anwser" for the question, I will select it as solver. – user1747036 Oct 20 '18 at 15:15
1

Think about redirecting the entire loop's stdout to the report.csv file, and, within, just echo as you did before. Like

for i in "${sqlanswer[@]}"; do
    ( echo somework #get the data from sql
      echo "$DDeviceID;$NBPort;$PortNumber;$CConnectedDevice;$Connectedport;$CabinetLocation;$HeightDevice;$PositionDevice;$ZoneName;$DataCenterName"
    ) &
  done > report.csv

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.