1

I have several files (*data.txt) and I am trying to split each of them into multiple files based on the content of column 1. I have managed to split them but I do not know how to name the output files with both $filename and column 1($1) using print. At the moment print in the following command gives me $1 ".txt", so for example: ENSG00000108094.txt, ENSG00000115232.txt instead of file1_ENSG00000108094.txt, file1_ENSG00000115232.txt which is not suitable as I need to have separate outputs for each input file. Here it is my command and I am not sure where I should use "$b" to get the expected outcome.

for filename in *_data.txt
 do
    b=${filename%%_data.txt}
cat $filename | awk 'NR==1 {header = $0; next}!header_printed[$1]++ {print header > $1".txt"}{print > $1".txt"}'
done

Thanks.

2
  • If you get rid of the uuoc and pass the filename to awk directly, you could derive b from the internal FILENAME variable. For only "several" files you could likely omit the loop and pass all the *_data.txt to a single invocation of awk and use FNR==1 in place of NR==1 to set the header and new filename - or in GNU awk, use the special BEGINFILE rule. Aug 24, 2021 at 12:49
  • @steeldriver. Thank you for your suggestion, but I do not fully understand your description. Could you please write the code you propose above?
    – user485085
    Aug 24, 2021 at 13:05

1 Answer 1

0

There are several ways to pass a shell variable into an awk program:

  1. using the -v command line option:

    awk -v b=${filename%data.txt} '... {print > (b $1 ".txt")}'
    
  2. pass the value as an ordinary argument after the awk program

    awk '... {print > (b $1 ".txt")}' b=${filename%data.txt}
    
  3. pass the value in the environment and access it within awk via the intenal ENVIRON array

    b=${filename%data.txt} awk '... {print > (ENVIRON["b"] $1 ".txt")}'
    

However, if you only have "several" files it might make sense to omit the shell loop altogether, and pass all the globbed files directly to awk, where you may derive the output file prefix from the FILENAME internal variable, ex.:

awk '
  FNR==1 {header = $0; b = FILENAME; sub(/data.txt$/,"",b); next}
  !header_printed[b $1]++ {print header > (b $1 ".txt")}
  {print > (b $1 ".txt")}
' *_data.txt

(you could also use split or substr to remove the data.txt suffix - I used regex sub because it is closest to the ${filename%data.txt} shell expansion).

1
  • Thank you very much. I have accepted your answer!
    – user485085
    Aug 24, 2021 at 16:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .