How to pass a variable into the field option for the cut command (bash)

Question

I have a tab delimited file with 2500 columns and 3000 rows. I would like to extract a specific column based on the header. By the way, this is being done in a while loop so each time a different column name will be selected.

Example file format:

a     b    c    d    e     f     g    h   
xy    1    2    f    21    4     5    6
qu    9    10   z    50    12    7    8

Desired output:

a     b    c    d    g 
xy    1    2    f    5
qu    9    10   z    7

I was thinking to identify the column number, and then store the number as a variable and pass it under -f in cut. Example if we are selecting column "g" which here is the 7th field.

colNum=$(head -1 file.txt | tr '\t' '\n' | cat -n | grep "g" | cut -f 1)
cut -f1,2,3,4,"$colNum" file.txt > new_file.txt

I get the following error:

cut: fields and positions are numbered from 1
Try 'cut --help' for more information.

Answer 1

In a comment, it was agreed that an answer doesn't need to involve cut. I would like to suggest that GNU Datamash (https://www.gnu.org/software/datamash/) solves this problem elegantly.

I put the sample input from the question into a file called data.txt. Then an example run processing it with datamash:

$ datamash -HW cut a,b,c,d,g < data.txt
cut(a)  cut(b)  cut(c)  cut(d)  cut(g)
xy      1       2       f       5
qu      9       10      z       7

If it's undesirable to add the name of the operation to the headers, a little post-processing can tidy it up a bit:

$ datamash -HW cut a,b,c,d,g < data.txt | sed 's/cut(\([^[:space:]]*\))/\1/g'
a       b       c       d       g
xy      1       2       f       5
qu      9       10      z       7

Answer 2

It is possible to extract a specific column from a tab delimited file using the 'cut' command. To do this, you need to identify the column number of the required column and then use the '-f' option with the cut command to specify the column number. For example, if you want to extract the 'g' column from your file, you can use the following command:

colNum=$(head -1 file.txt | tr '\t' '\n' | cat -n | grep "g" | cut -f 1)
cut -f1,2,3,4,"$colNum" file.txt > new_file.txt

The first command identifies the column number of the desired column:

colNum=$(head -1 file.txt | tr '\t' '\n' | cat -n | grep "g" | cut -f 1)

The second command uses the 'cut' command to extract the specified column from the file:

cut -f1,2,3,4,"$colNum" file.txt > new_file.txt.

This should extract the desired column from the file and save it in the new_file.txt. For more information on the 'cut' command, please consult the official documentation here: https://linux.die.net/man/1/cut.

As far as I can see error message you're seeing is indicating that the cut command expects its field argument to be a number, but you're passing in a variable. To fix this, you can remove the quotes around "$colNum" in the cut command, so that the variable is interpreted as a number rather than a string.

Here's the corrected command:

cut -f1,2,3,4,$colNum file.txt > new_file.txt

Alternatively, you can use awk command to extract the specific column based on the header.

awk -v col="g" -F'\t' 'NR==1{for(i=1;i<=NF;i++){if($i==col){c=i;break}}} {print $1,$2,$3,$4,$c}' file.txt > new_file.txt

This command will look for the column header "g" in the first row and saves the column number in variable "c" and prints the first four columns and the column with header "g" in the new_file.txt.