2

I have to subset a file for different numbers in a column and create several files in output. I am trying to do this in a loop but some how the iterator is not working.

Input (tab-separated fields)

abc 1  
aaa 1  
ccc 1  
asd 2  
sad 2  
aaf 3  

Output

File1:

abc 1  
aaa 1  
ccc 1    

File2:

asd 2  
sad 2

File3:

aaf 3  

My test is this but it outputs only empty files:

for i in $(seq 1 3); do awk -F "\t" '{$2 == $i}' input  > cluster.$i.txt; done 
3
  • Are your input fields tab-separated? (Your awk code seems to expect them to be.) – roaima Aug 1 '19 at 18:53
  • yes they are tab separated, have you tried if it works for you? – fusion.slope Aug 1 '19 at 18:54
  • Thanks. I couldn't test without knowing the input format – roaima Aug 1 '19 at 19:01
2

The reason your code is failing is that you're trying to use a shell variable $i inside an awk script but doing so incorrectly. Text inside single quotes '...' is treated literally by the shell so the $i is seen as two characters rather than the value of the shell variable $i. You're also trying to perform a comparison as an action (which means there's no implied action to print the line).

You could pass the value in to awk

awk -F $'\t' -v i="$i" '$2 == i' input > "cluster.$i.txt"

Or you could treat each line on its own merit and avoid awk entirely:

while read field index
do
    printf "%s\t%s\n" "$field" "$index" >> "cluster.$index.txt"
done < input

Or you could use awk for much the same thing:

awk '{ fname = "cluster." $2 ".txt"; print > fname }' input
8
  • 1
    There are buggy awk implementations out there that don't allow you to redirect to an expression like that. The solution is simply to calculate the output filename first, and then redirect to that: fname = "cluster." $2 ".txt"; print >fname. – Kusalananda Aug 1 '19 at 19:09
  • 1
    @Kusalanander I haven't come across such implementations myself but am happy to accept your advice. Thank you. – roaima Aug 1 '19 at 21:26
  • 2
    @Kusalananda those implementations aren't buggy, it's just that an unparenthesized expression on the right side of input or output redirection is undefined behavior per POSIX. So print > "foo" 7 can be interpreted by any given awk as (print > "foo") (7) (a syntax error) or print > ("foo" 7) (the presumably desired behavior and still be POSIX compliant. So you can put an expression on the right side of in/out redirection with > or < but put it in parens if you want it to work in all awks. – Ed Morton Aug 1 '19 at 21:57
  • 1
    @roaima that last awk script is the right approach but as written it'll fail with a "too many open files" error message once you get past about a dozen output file names unless you're using gawk which handles that internally. You can solve the problem for all awks with awk '$2!=prev{ close(fname); prev=$2; fname = "cluster." $2 ".txt"} {print > fname }' input for sorted input or awk '{ fname = "cluster." $2 ".txt"; print >> fname; close(fname) }' input otherwise. – Ed Morton Aug 1 '19 at 22:00
  • 1
    @fusion.slope the solution you've chosen involving a shell loop will be orders of magnitude slower than the final 1-call-to-awk solution above. – Ed Morton Aug 1 '19 at 22:03
2

$i inside the awk expression doesn't refer to shell variable i. You can pass the variable in on the command line using -v i="$i" and refer to it as plain i.

Also {....} defines an action not a pattern; you appear to want to perform the default action (print) when the pattern $2 == i is true. So

for i in $(seq 1 3); do 
  awk -F "\t" -v i="$i" '$2 == i' 
input  > cluster.$i.txt; done

However, rather than calling awk once for every index, you might consider doing something like

awk -F "\t" '{print > "cluster" $2 ".txt"}' input

which uses the input's $2 directly to construct the output file name.

Note that unless you are specifically trying to prevent field-splitting on space characters, you can probably drop the -F "\t" and allow awk to split on the default whitespace separators.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.