Storing output of awk to an array and print it to a file with comma delimiters

Question

I am trying to use bash to go through multiple directories (sims) and search for a given string, setting each index of an array to its relevant output, then print that output with comma delimiters for each value and adding a semicolon delimiter to separate each file. Effectively, this should give me a CSV that I can split twice in excel. All the important information is in field 3 each time it finds "Total Energy", which is found in the file named "output" in each directory.

With the set of directories that I am currently testing, each output file should have 2500 energies, but the code currently does not find that (vide infra).

Here is the code I have so far:

#/bin/bash/

saveIFS="$IFS"

#Step 1: Ask user for the range of sims they want
echo "What is the first sim?"
read simcount
echo "What is the last sim?"
read simend

#Step 2: Create the energy files with proper naming conventions and make sure they're empty
energies+="energies${simcount}-${simend}.csv"
fenergies+="final_energies${simcount}-${simend}.out"
touch $energies
touch $fenergies
< $energies
< $fenergies

#Step 3: Go through each directory, print all energies into proper files
while [ $simcount -le $simend ]; do
        echo $simcount
        cd $print'sim'$simcount                                 # Change to the directory of each specified sim
        energy=($(awk '/Total Energy/{ print $3 }' output))     # Print all energies from output into an array
        echo ${#energy[@]}
        fenergy=${energy[${#energy[@]}-1]}                      # Get the last energy in each file
        cd ../                                                  # Go up a directory
        IFS=","                                                 # Change the Internal Field Separator (IFS) to a comma
        echo "${energy[*]};" >> $energies                       # Expand the array of energies into an IFS-delimited list; print it into the new energies file
        echo "$fenergy" >> $fenergies                           # Put the final energy of each sim on a new line in the new final energies file
        ((simcount++))
done

IFS="$saveIFS"

exit 0

This gives the following output:

$ e.sh
What is the first sim?
6
What is the last sim?
15
6
2500
7
1
8
1
9
1
10
1
11
1
12
1
13
1
14
1
15
1

Which means that the loop is finding all 2500 energies the first time around, but each subsequent time through the loop it does not split the output of awk into an array. A representative example of the output this puts in the new file denoted as $energies:

-271.2872230353,-271.3198859908,-271.4166545741,-271.5362409096,-271.6700236287,-271.8068505329,-271.9076587286,...;
-273.2853761106
-273.2855419371
...
-273.2856368361
-273.2857720402
-273.2859963834;
-271.2872230353
-271.3198859908
-271.4166545741
...

To clarify, the first iteration of the loop is successful and outputs the array onto one line with a semicolon separator. All following iterations do not get split into an array (or have an array length of 1) and seem to repeat thousands of times before moving onto the next directory.

I have searched around for a while now, and I cannot figure out why this would happen. I have also tried unsetting energies at the end of each iteration to no avail. So my specific question(s) is(are): why would splitting awk output into an array work the first time through the loop but not any subsequent times? Are there better/more efficient ways of going about this using bash that would be worth checking out?

Answer 1

The first time through your loop you set energy=( $(awk ...) ) and the output from awk consists of several lines each containing one number and terminated by a newline character, as lines in Unix always are. Command substitution $( ... ) strips trailing newline(s), and when it isn't in double-quotes, then breaks the result into 'words' at any of space tab newline (you have newlines), and finally if any word is a 'pattern' (containing ?*[..]) that matches any filename(s) it is replaced by those filename(s) as separate 'words' (you have no such 'glob' patterns). Array assignment energy=( ... ) then stores those words as elements of the array.

The second time through IFS is set to comma. Now when $( ... ) tries to split into words it uses (only) comma and there are no commas in the output from awk, so the entire output (including the newlines) is left as one word, and is assigned to the array as one element.

You need to revert IFS for each iteration. And further you need to have IFS set to its standard value, or at least a value containing newline, on entry to this script. OTOH reverting IFS just before exiting the script is mostly useless; a script is normally run in a separate shell process and any variable settings or other in-process changes made by the script are discarded when the script exits.

Or, instead of changing IFS and explicitly reverting, you can do it in a subshell so that the change is discarded when the subshell completes. The shell syntax for subshell is also parentheses, this time by themselves:

( IFS=","; echo "${energy[*]};" >> $energies )
# you don't actually need to quote , here but 
# it's a good habit for string literals in general

Also in general printf is safer than echo because depending on the shell and/or system you use echo can mangle some string values. However, the values you have here -- restricted to decimal numbers -- are safe for echo.

For bash, another possibility is to treat the data just as a single string, not an array:

energy=$( awk '/Total energy/{print $3}' output )
# command substitution strips the last newline
# scalar assignment does NOT do wordsplit and glob 
echo "${energy//$'\n'/,};" >>energies_blah 
# replaces all other newlines with commas, and adds semicolon 
echo "${energy##*$'\n'}" >>final_energies_blah 
# removes everything up to and including the last newline, 
# leaving only the last number

Or you could actually do the whole job with awk, especially non-ancient GNU awk with 'endfile':

# read simcount,simend and set energies,fenergies
infiles=$( printf 'sim%d/output ' $( seq $simcount $simend ) )
awk -vf1=$energies -vf2=$fenergies '/Total Energy/ {e=e","$3; f=$3} ENDFILE {print substr(e,2)";">>f1; print f>>f2; e=f=""}' $infiles

With other awk's you can accomplish the same thing with slightly uglier code by (first!) checking FNR==1&&NR>1 for the end of each file other than the last and (anywhere) END for the end of the last file.