I have a file with a large number of space separated columns. I want to print specific columns based on certain numerical criteria in a dynamic way. For example:

]$ cols=$(for i in `seq 1 3`; do echo -n "\$$[$[i-1]*6+1],\$$[$[i-1]*6+2],\$$[$[i-1]*6+3],\$$[$[i-1]*6+4+66],\$$[$[i-1]*6+5+66],\$$[$[i-1]*6+6+66],"; done)

which gives me the columns I want to print out:

]$ echo ${cols%?}

when I pass this to awk as a string, I don't get what I want:

]$ awk -v cols=${cols%?} '{print cols}' file-testawk | head -2

awk treats it as a string rather than column identifiers.

How can I pass a string of columns to print out to awk in a way that will be correctly recognized? I'm looking for a simple, more-or-less one-liner solution, e.g. like this:

cols=$(for i in `seq 1 3`; do echo -n "\$$[$[i-1]*6+1],\$$[$[i-1]*6+2],\$$[$[i-1]*6+3],\$$[$[i-1]*6+4+66],\$$[$[i-1]*6+5+66],\$$[$[i-1]*6+6+66],"; done); awk -v cols=${cols%?} '{print cols}' file-testawk > file.out

awk does not have eval like capabilities but you can make a trick using awk -f capability (read script from file) combined with bash process substitution:

$ a="\$1,\$4"
$ echo "$a"
$ a="{print $a}"
$ echo "$a"
{print $1,$4}
$ awk -f <(echo "$a") <<<"one two three four five"
one four
  • Beautiful. awk -f <(echo "{print ${cols%?}}") file-testawk | head -2 works like a charm. – Dimi Jun 2 '17 at 20:19

Usage: ./pass_numbers_to_awk.sh Explanation in comments.


#generate random string of numbers - simulation column's numbers 
for i in {1..2}; do
    for j in {1..3}; do
        num=$(( (i-1) * 6 + j ))
        #numbers separated by vertical bar symbol 

# pass to awk string like a "1|2|3|7|8|9|13|14|15|", 
# removing last vertical bar "|"
# use the awk split function - for information 
# look at the 'man mawk | grep -A 3 split\(s,A,r\)'
# go through array and print specified columns.

awk -v string_from_bash="${string_of_numbers%?}" '
    num_of_cols = split(string_from_bash, array_of_columns, "|");
    for (i = 1; i <= num_of_cols; i++) {

        # Prevent trailing spaces emergence
        OFS = (i > 1) ? " " : ""

        printf "%s%s", OFS, $array_of_columns[i];
    printf "\n";
}' < input.txt

Creating the input.txt file for testing: ./create_table.sh > input.txt


for i in {A..O}; do
    for j in {1..10}; do
        echo -n "column_${j} "
  • 1
    awk for(idx in ary) is allowed to enumerate idx values in effectively random order, and almost always does, which garbles your output. gawk 4 has a nondefault option to sort them; for(i=1; i in ary; i++) ... ary[i] ... or n=split(,ary,); ... for(i=1; i<=n; i++) ...ary[i] ... work portably. – dave_thompson_085 Jun 4 '17 at 7:11
  • @dave_thompson_085 Thanks, didn't know about it. Changed the code. – MiniMax Jun 4 '17 at 18:47

awk is adept at doing these index calculations, so this:

awk -v N=3 '
   for ( i=1; i<= N; ++i )
      print $((i-1)*6+1), $((i-1)*6+2), $((i-1)*6+3), $((i-1)*6+4+66), $((i-1)*6+5+66), $((i-1)*6+6+66)
' data.file

Basic idea being, if you give awk a number stored in a variable i then awk can fetch the field corresponding to that number via $(i). Now i can as well be an expression also as is the case here.

  • This is good, but it is not very dynamic. Basically, I don't want to do anything in awk because for i in `seq 1 3` can change randomly, e.g. for i in 2 4 6 13 21 as well as the functional dependences on $i. – Dimi Jun 2 '17 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.