multiple pipes in loop, saving pipeline-result to array

Question

I am trying to do the following (using bash): Search for files that always have the same name and extract data from these files. I want to store the extracted data in new arrays I am almost there, I think, see code below.

The files I am searching for all have this format:

 #!/bin/bash
  echo "the concentration of NDPH is 2 mM, which corresponds to 2 molecules in a box of size 12 nm (12 x 12 x 12 nm^3)" > README_test

#find all the README* files and save the paths into an array called files
  files=()
  data1=()
  data2=()
  data3=()

  while IFS=  read -r -d $'\0'; do
files+=("$REPLY")
  #open all the files and extract data from them
  while read -r line
  do
name="$line"
echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}' 
echo "$name" 
echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}'
data1+=( "$echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}' )" )    

# variables are not preserved...
# data2+= echo "$name"  | tr ' ' '\n'|  awk 'f{print;f=0;exit} /is/{f=1}'
echo "$name"  | tr ' ' '\n'|  awk 'f{print;f=0;exit} /size/{f=1}'
# variables are not preserved... 
# data3+= echo "$name"  | tr ' ' '\n'|  awk 'f{print;f=0;exit} /size/{f=1}'
  done < "$REPLY"
  done < <(find . -name "README*" -print0)
  echo ${data1[0]}

The issue is that the pipe giving me the exact output I want from the files is "not working" (variables are not preserved) in the loops. I have no idea how/if I can use process substitution to get what I want: an array (data1, data2, data3) filled with the output of the pipes.

UPDATE: SO I was not assigning things to the array correctly (see data1, which is properly assigning sth now.) But why are

echo ${data1[0]}

and

echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}'

not the same?

SOLUTION (as per ilkkachu' s accepted answer):

  #!/bin/bash
  echo "the concentration of NDPH is 2 mM, which corresponds to 2 molecules in a box of size 12 nm (12 x 12 x 12 nm^3)" > README_test
  files=()
  data1=()
  data2=()
  data3=()

  get_some_field() {    
 echo "$1" | tr ' ' '\n'|  awk -vkey="$2" 'f{print;f=0;exit} $0 ~ key {f=1}' 
  }

  #find all the README* files and save the paths into an array called files
  while IFS=  read -r -d $'\0'; do
files+=("$REPLY")
  #open all the files and extract data from them
  while read -r line
  do
name="$line"
echo "$name" 
echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}'
data1+=( "$(get_some_field "$name" of)" )
data2+=( "$(get_some_field "$name" is)" )
data3+=( "$(get_some_field "$name" size)" )

  done < "$REPLY"
 done < <(find . -name "README*" -print0)

  echo ${data1[0]}
  echo ${data2[0]}
  echo ${data3[0]}

Answer 1

I'm assuming you want the output of the echo ... | awk stored in a variable, and in particular, appended to one of the arrays.

First, to capture the output of a command, use "$( cmd... )" (command substitution). As a trivial example, this prints your hostname:

var=$(uname -n)
echo $var

Second, to append to an array, you need to use the array assignment syntax, with parenthesis around the right hand side. This would append the value of var to the array:

array+=( $var )

And third, the expansion of $var and the command substitution $(...) are subject to word splitting, so you want to use parenthesis around them. Again a trivial example, this puts the full output of uname -a as a single element in the array:

array+=( "$(uname -a)" )

Or, in your case, in full:

data1+=( "$(echo "$1" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}')" )

_{(Note that the quotes inside the command substitution are distinct from the quotes outside it. The quote before $1 doesn't stop the quoting started outside $(), unlike what the syntax hilighting on SE seems to imply.)}

You could make that slightly simpler to read by putting the pipeline in a function:

get_data1() {
    echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}'
}
...
data1+=( "$(get_data1)" )

Or, as the pipelines seem similar, use the function to avoid repeating the code:

get_some_field() {
    echo "$1" | tr ' ' '\n'|  awk -vkey="$2" 'f{print;f=0;exit} $0 ~ key {f=1}'
}

and then

data1+=( "$(get_some_field "$name" of)" )
data2+=( "$(get_some_field "$name" is)" )
data3+=( "$(get_some_field "$name" size)" )

_{(If I read your pipeline right, that is, I didn't test the above.)}