3

The question is really simple, I have read all the questions and still cannot make it! I have a plain file like this

$cat file1.txt
ALA
AJD
KSF

And I want awk to use each of the values as regex to print lines from another file to another one:

$cat file2.txt
name,st,ed,le
ALA,10,12,12
ALA,2,5,4
ALA,6,5,8
AJD,5,8,7
KSF,5,8,7

So my script is

while read p;
awk -F"," 'NR==1{print $0}' file2.txt > $p.csv
awk -F"," '/$p/{print $0}' file2.txt >> $p.csv
done <file1.txt

And the desired output would be:

$cat ALA.csv
name,st,ed,le
ALA,10,12,12
ALA,2,5,4
ALA,6,5,8
$cat AJD.csv
name,st,ed,le
AJD,5,8,7
$cat KSF.csv
name,st,ed,le
KSF,5,8,7

Unfortunately, I only get the headers printed in each file. I have manually put each value from file1.txt replacing $p and it works perfectly. So I think the problem is that the variable $p is not been well interpreted. I tried with quotes, double simple. I tried also many different suggestions I found, but nothing seems to work!

4

While you could do:

awk "/$p/" file2.txt > "$p.csv"

That is have the shell expand the content of the $p shell variable in the code passed to awk, that's bad practice and basically amounts to a command injection vulnerability (for instance for a value of $p like ^/{system("reboot");/). Best is to pass the shell variable as is to awk and use awk's ~ operator for regexp matching. The best way is via an environment variable and awk's ENVIRON special array:

export P
while IFS= read -r P; do
  awk 'NR == 1 || $0 ~ ENVIRON["P"]' < file2.txt > "$P.csv"
done < file1.txt

But here, you could avoid the shell loop and do only one pass in the files:

awk 'NR == FNR {files[$0]; next}
     FNR == 1 {for (f in files) print > f ".csv"; next}
     {
       for (f in files)
         if ($0 ~ f) print > f ".csv"
     }' file1.txt file2.txt
  • Is there really a difference if the variable is passed through the environment or via -vP="$P"? Can the second one choke on some weird special character? – ilkkachu Oct 28 '16 at 11:09
  • @ikkachu, yes -vP="$P" does some backslash sequence expansion (typically a problem with regexps that often contain backslashes). See unix.stackexchange.com/search?q=user%3A22565+ENVIRON+ARGV+awk – Stéphane Chazelas Oct 28 '16 at 11:23
  • By the way -vP="$P" doesn't work in many traditional implementations of awk where you need -v P="$P".(-v and P="$P" being separate arguments). – Stéphane Chazelas Oct 28 '16 at 11:26
  • Much memory save if you iterate through fields not through array {for(i=1;i<=NF;i++)if($i in files)print > $i ".csv" (do not forget FS=OFS=",") – Costas Oct 28 '16 at 11:41
  • @Costas, that would be functionally different. I'm not sure how that would save memory though. And you could write the same line several times into a given file. – Stéphane Chazelas Oct 28 '16 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.