0

I am trying to do the following (using bash): Search for files that always have the same name and extract data from these files. I want to store the extracted data in new arrays I am almost there, I think, see code below.

The files I am searching for all have this format:

 #!/bin/bash
  echo "the concentration of NDPH is 2 mM, which corresponds to 2 molecules in a box of size 12 nm (12 x 12 x 12 nm^3)" > README_test

#find all the README* files and save the paths into an array called files
  files=()
  data1=()
  data2=()
  data3=()

  while IFS=  read -r -d $'\0'; do
files+=("$REPLY")
  #open all the files and extract data from them
  while read -r line
  do
name="$line"
echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}' 
echo "$name" 
echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}'
data1+=( "$echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}' )" )    

# variables are not preserved...
# data2+= echo "$name"  | tr ' ' '\n'|  awk 'f{print;f=0;exit} /is/{f=1}'
echo "$name"  | tr ' ' '\n'|  awk 'f{print;f=0;exit} /size/{f=1}'
# variables are not preserved... 
# data3+= echo "$name"  | tr ' ' '\n'|  awk 'f{print;f=0;exit} /size/{f=1}'
  done < "$REPLY"
  done < <(find . -name "README*" -print0)
  echo ${data1[0]}

The issue is that the pipe giving me the exact output I want from the files is "not working" (variables are not preserved) in the loops. I have no idea how/if I can use process substitution to get what I want: an array (data1, data2, data3) filled with the output of the pipes.

UPDATE: SO I was not assigning things to the array correctly (see data1, which is properly assigning sth now.) But why are

echo ${data1[0]}

and

echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}'

not the same?

SOLUTION (as per ilkkachu' s accepted answer):

  #!/bin/bash
  echo "the concentration of NDPH is 2 mM, which corresponds to 2 molecules in a box of size 12 nm (12 x 12 x 12 nm^3)" > README_test
  files=()
  data1=()
  data2=()
  data3=()

  get_some_field() {    
 echo "$1" | tr ' ' '\n'|  awk -vkey="$2" 'f{print;f=0;exit} $0 ~ key {f=1}' 
  }

  #find all the README* files and save the paths into an array called files
  while IFS=  read -r -d $'\0'; do
files+=("$REPLY")
  #open all the files and extract data from them
  while read -r line
  do
name="$line"
echo "$name" 
echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}'
data1+=( "$(get_some_field "$name" of)" )
data2+=( "$(get_some_field "$name" is)" )
data3+=( "$(get_some_field "$name" size)" )

  done < "$REPLY"
 done < <(find . -name "README*" -print0)

  echo ${data1[0]}
  echo ${data2[0]}
  echo ${data3[0]}
  • data1+= echo... doesn't really do anything to the data1 variable. Do you mean to use data1+=( "$(echo ... | awk)" ) ? – ilkkachu Jul 25 '18 at 10:20
  • @ilkkachu yes indeed, you are correct. – gugy Jul 25 '18 at 10:23
  • "The files I am searching for all have this format:" - Did you mean to include an example of the files you are trying to process? – steeldriver Jul 25 '18 at 10:59
  • @steeldriver because there are many files I am only giving you one example, written in the first line to the file README_test (the files are all one-liner outputs from another program) – gugy Jul 25 '18 at 12:20
2

I'm assuming you want the output of the echo ... | awk stored in a variable, and in particular, appended to one of the arrays.

First, to capture the output of a command, use "$( cmd... )" (command substitution). As a trivial example, this prints your hostname:

var=$(uname -n)
echo $var

Second, to append to an array, you need to use the array assignment syntax, with parenthesis around the right hand side. This would append the value of var to the array:

array+=( $var )

And third, the expansion of $var and the command substitution $(...) are subject to word splitting, so you want to use parenthesis around them. Again a trivial example, this puts the full output of uname -a as a single element in the array:

array+=( "$(uname -a)" )

Or, in your case, in full:

data1+=( "$(echo "$1" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}')" )

(Note that the quotes inside the command substitution are distinct from the quotes outside it. The quote before $1 doesn't stop the quoting started outside $(), unlike what the syntax hilighting on SE seems to imply.)

You could make that slightly simpler to read by putting the pipeline in a function:

get_data1() {
    echo "$name" | tr ' ' '\n'|  awk 'f{print;f=0;exit} /of/{f=1}'
}
...
data1+=( "$(get_data1)" )

Or, as the pipelines seem similar, use the function to avoid repeating the code:

get_some_field() {
    echo "$1" | tr ' ' '\n'|  awk -vkey="$2" 'f{print;f=0;exit} $0 ~ key {f=1}'
}

and then

data1+=( "$(get_some_field "$name" of)" )
data2+=( "$(get_some_field "$name" is)" )
data3+=( "$(get_some_field "$name" size)" )

(If I read your pipeline right, that is, I didn't test the above.)

  • this works - and the explanations helped me to understand what went wrong :) – gugy Jul 25 '18 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.