2

I'll appreciate your help with the following issue:

I'm trying to set an array which contains a variable as part of the array name, example: Arr_$COUNTER (where $COUNTER is changed based on a loop count)

Every possible way I have tried came up with an error, such as "bad substitution" or "syntax error near unexpected token"

Here is the entire flow:

  1. There is one file which contain multiple lines. each line has 6 values separated by space

    10 20 30 40 50 60  
    100 200 300 400 500 600
    
  2. The script, is meant to read each line from the file, and declare it as an array (with the line number which is the variable.

  3. as a test, each value should be printed and eventually another function will be executed on each value.

    #!/bin/bash
    COUNTER=1
    LINES=`wc -l VALUES_FILE.txt | awk '{print $1}'`
    echo "Total number of lines "$LINES
    echo
    while [ $COUNTER -le $LINES ]
    do
    echo "Counter value is $COUNTER"
    field=`awk "NR == $COUNTER" VALUES_FILE.txt`
    echo "Field = $field"
    declare -a "arr$COUNTER=($field)"
    echo "arr$COUNTER[0] = ${arr$COUNTER[0]}"
    echo "arr$COUNTER[1] = ${arr$COUNTER[1]}"
    echo "arr$COUNTER[2] = ${arr$COUNTER[2]}"
    echo "arr$COUNTER[3] = ${arr$COUNTER[3]}"
    echo "arr$COUNTER[4] = ${arr$COUNTER[4]}"
    echo "arr$COUNTER[5] = ${arr$COUNTER[5]}"
    let COUNTER=COUNTER+1
    echo
    done
    echo "The End"
    echo
    

Here is the result:

Total number of lines 2

Counter value is 1
Field = 10 20 30 40 50 60
./sort.sh: line 12: arr$COUNTER[0] = ${arr$COUNTER[0]}: bad substitution
The End

What should be changed / fixed in order to have it working properly?

thank !

1

Some ideas:

  1. A "parameter expansion" of a variable value (the ${...} part):

    echo "arr$COUNTER[0] = ${arr$COUNTER[0]}"
    

    will not work. You may get around by using eval (but I do not recommend it):

    eval echo "arr$COUNTER[0] = \${arr$COUNTER[0]}"
    

    That line could be written as this:

    i="arr$COUNTER[0]"; echo "$i = ${!i}"
    

    That is called indirection (the !) in Bash.

  2. A similar issue happens with this line:

    declare -a "arr$COUNTER=($field)"
    

    Which should be split into two lines, and eval used:

    declare -a "arr$COUNTER"
    eval arr$COUNTER\=\( \$field \)
    

    Again, I do not recommend using eval (in this case).

  3. As you are reading the whole file into the memory of the shell, we may as well use a simpler method to get all lines into an array:

    readarray -t lines <"VALUES_FILE.txt"
    

    That should be faster than calling awk for each line.

An script with all the above could be:

#!/bin/bash
valfile="VALUES_FILE.txt"

readarray -t lines <"$valfile"             ### read all lines in.

line_count="${#lines[@]}"
echo "Total number of lines $line_count"

for ((l=0;l<$line_count;l++)); do
    echo "Counter value is $l"             ### In which line are we?
    echo "Field = ${lines[l]}"             ### kepth only to help understanding.
    k="arr$l"                              ### Build the variable arr$COUNTER
    IFS=" " read -ra $k <<<"${lines[l]}"   ### Split into an array a line.
    eval max=\${#$k[@]}                    ### How many elements arr$COUNTER has?
    #echo "field $field and k=$k max=$max" ### Un-quote to "see" inside.
    for ((j=0;j<$max;j++)); do             ### for each element in the line.
        i="$k[$j]"; echo "$i = ${!i}"      ### echo it's value.
    done
done
echo "The End"
echo

However, still, AWK may be faster, if we could execute what you need in AWK.


A similar processing could be done in awk. Assuming the 6 values will be used as an IP (4 of them) and the other two are a number and an epoch time.

Just a very simple sample of an AWK script:

#!/bin/sh
valfile="VALUES_FILE.txt"
awk '
NF==6 { printf ( "IP: %s.%s.%s.%s\t",$1,$2,$3,$4)
        printf ( "number: %s\t",$5+2)
        printf ( "epoch: %s\t",$6)
        printf ( "\n" )
    }
' "$valfile"

Just make a new question with the details.

  • Thank you for the detailed information - that helped ! unfortunately the "readarray" is not working for me .. – AlonCo Feb 17 '16 at 11:10
  • @AlonCo Assuming readarray is not available because of your bash version, then take a look to the detailed analysis in here. I am sure you will find a solution there. – user79743 Feb 18 '16 at 4:28
  • my script is completed thanks to you and as you already guessed it's SLOW. You mentioned AWK can be faster, how? The script eventually will be used to read a large file ( 100k lines) where each line has 6 values and convert these values to IP / number/ epoch time. – AlonCo Feb 18 '16 at 5:07
  • @AlonCo What is the intended processing needed for each file line?. If that could be done in AWK, then an awk script will run quite faster. Could you reveal the intended processing or raise a new question with the details? – user79743 Feb 18 '16 at 6:26
  • @AlonCo I added an AWK script to only print the values to my answer. Take a look, time it. It does not read the whole file into memory (which should be an additional plus IMO). – user79743 Feb 18 '16 at 6:58
2

You can use the variable indirection, if you assign both the name and index to a variable:

s="arr$COUNTER[0]"
echo "arr$COUNTER[0] = ${!s}"
  • I'm not sure I get it. Can you explain further? – AlonCo Feb 15 '16 at 22:39
  • Variable indirection ${!s} works with arrays only if you use both the array name and index. See "variable indirection" under "Parameter Expansion" in man bash. – choroba Feb 15 '16 at 23:04
1

You can generate names using eval, e.g.,

eval declare -a '"arr'$COUNTER'=($field)"'

essentially quoting all of the meta-characters except the ones you want to evaluate.

So... if $COUNTER is 1, your script would do

declare -a "arr1=($field)"

Further reading:

  • This might work... I'll test it and update. Thanks?! – AlonCo Feb 15 '16 at 22:38
  • eval is POSIX (works "everywhere"), but keep in mind that all of the generated names would get similar treatment. Readability of the script is a problem with either eval or variable indirection. – Thomas Dickey Feb 15 '16 at 22:42
1

The standard way of storing multi-dimensional array data in an array of dimension 1 is to store each row at an offset into the array.

Element (i,j) will be located at index i*m + j where i is the zero-based row index, j is the zero-based column index, and m is the number of columns.

This also makes it easier to read the data in as we can just take your input file, change all spaces to newlines and use readarray.

On the command line:

$ readarray -t arr < <( tr -s ' ' '\n' <data )
$ printf '%s\n' "${arr[@]}"
10
20
30
40
50
60
100
200
300
400
500
600

We can figure out the number of columns in the data with

$ m=$( awk '{ print NF; exit }' <data )

And the number of rows:

$ n=$( wc -l <data )

Then we may iterate over columns and rows in a double loop as usual:

for (( i = 0; i < n; ++i )); do
    for (( j = 0; j < m; ++j )); do
        printf '%4d' "${arr[i*m + j]}"
    done
    printf '\n'
done

For the given data, this would generate

  10  20  30  40  50  60
 100 200 300 400 500 600

Ideally, you'd use a language, such as Perl or Python or C, that supports multi-dimensional arrays. That is, if you actually need to store the whole set of data in memory at all and can't process it on a row by row basis.

For row by row processing, awk would be a good candidate to replace bash (any language would be a good candidate to replace a shell for any sort of data processing):

awk '{ for (i = 1; i <= NF; ++i) printf("%4d", $i); printf("\n") }' data

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.