3

About my code:

  1. get all .txt files
  2. for each file, execute the awk command. All files are processed in parallel
  3. if colum 3 does NOT contain "needle", write line to file

Result:

-bash: !~: event not found

Code:

ls *.txt | parallel "awk -F '\t' -v OFS='\t' '$1 { if($3 !~ /needle/){print;} ;}' {} > {}.output"

Question:

  • What am I doing wrong?

Thank you

2

This looks like a quoting issue. Perhaps the easiest way to debug this is to pass the option --dry-run to parallel:

$ parallel --dry-run "awk -F '\t' -v OFS='\t' '$1 { if($3 !~ /needle/){print;} ;}' {} > {}.output" ::: in
awk -F '\t' -v OFS='\t' ' { if( ~ /needle/){print;} ;}' in > in.output

There you can see that your variables $1 and $3 have disappeared, and that !~ has been expanded to ~ (in my case). All those have special meaning for the bash shell, and should be protected if they are meant for awk instead. Try the following instead (I also altered the awk script to fit your description):

parallel awk -F '\\t' -vOFS='\t' $'\'($3 !~ /needle/) { print }\'' \>{}.output {} ::: *.txt

The reason why those expansions happened is that the tokens were inside double quotes (they are within single quotes within double quotes, but the shell only concerns itself with the outer quotes).

Note that I also avoid parsing the output of ls.

0

When quoting gets too tough consider using bash functions:

myfunc() {
  awk -F '\t' -v OFS='\t' '$1 { if($3 !~ /needle/){print;} ;}' "$1" > "$1".output
}
export -f myfunc

parallel myfunc {} ::: *.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.