The question is really simple, I have read all the questions and still cannot make it! I have a plain file like this

$cat file1.txt

And I want awk to use each of the values as regex to print lines from another file to another one:

$cat file2.txt

So my script is

while read p;
awk -F"," 'NR==1{print $0}' file2.txt > $p.csv
awk -F"," '/$p/{print $0}' file2.txt >> $p.csv
done <file1.txt

And the desired output would be:

$cat ALA.csv
$cat AJD.csv
$cat KSF.csv

Unfortunately, I only get the headers printed in each file. I have manually put each value from file1.txt replacing $p and it works perfectly. So I think the problem is that the variable $p is not been well interpreted. I tried with quotes, double simple. I tried also many different suggestions I found, but nothing seems to work!


While you could do:

awk "/$p/" file2.txt > "$p.csv"

That is have the shell expand the content of the $p shell variable in the code passed to awk, that's bad practice and basically amounts to a command injection vulnerability (for instance for a value of $p like ^/{system("reboot");/). Best is to pass the shell variable as is to awk and use awk's ~ operator for regexp matching. The best way is via an environment variable and awk's ENVIRON special array:

export P
while IFS= read -r P; do
  awk 'NR == 1 || $0 ~ ENVIRON["P"]' < file2.txt > "$P.csv"
done < file1.txt

But here, you could avoid the shell loop and do only one pass in the files:

awk 'NR == FNR {files[$0]; next}
     FNR == 1 {for (f in files) print > f ".csv"; next}
       for (f in files)
         if ($0 ~ f) print > f ".csv"
     }' file1.txt file2.txt
  • Is there really a difference if the variable is passed through the environment or via -vP="$P"? Can the second one choke on some weird special character?
    – ilkkachu
    Oct 28 '16 at 11:09
  • @ikkachu, yes -vP="$P" does some backslash sequence expansion (typically a problem with regexps that often contain backslashes). See unix.stackexchange.com/search?q=user%3A22565+ENVIRON+ARGV+awk Oct 28 '16 at 11:23
  • By the way -vP="$P" doesn't work in many traditional implementations of awk where you need -v P="$P".(-v and P="$P" being separate arguments). Oct 28 '16 at 11:26
  • Much memory save if you iterate through fields not through array {for(i=1;i<=NF;i++)if($i in files)print > $i ".csv" (do not forget FS=OFS=",")
    – Costas
    Oct 28 '16 at 11:41
  • @Costas, that would be functionally different. I'm not sure how that would save memory though. And you could write the same line several times into a given file. Oct 28 '16 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.