Two files: data1

    Name       |formula           |no. |dose|days|cost  |msg|em|notes
    Fname-Lname|BXXXT+GG          |8262|4   |14  |57.78 |   |  |sq
    Fname-Lname|SJXXT+GG          |8263|4¾  |14  |105.15|   |  |IB
    Fname-Lname|FJDHT+BH,LG,CQC,ZX|8264|5¾  |14  |46.20 |   |  |IB



I am using the code snippet

awk 'BEGIN {FS = "|" } NR==FNR{a[$6];next} $3 in a {print $0}' data1 data2

To match where a value in $6 of file data1 also occurs in $3 of file data2. Where there is a match print out the whole record ($0) containing the match from file data2. I am expecting:


But I am only getting an output of a blank line. I removed the file separators "|" using a " " as replacement the command code worked exactly as expected however really want to preserve the field separator as | if at all possible . I would like to understand why the addition of a BEGIN block has caused this . Has it caused awk to load an empty array in place of taking data from S6 ? My awk level is just above beginner. Edit: I have also used the -F parameter with the same result, an out put of a blank line . I am using gawk .

  • 1
    what about if you do awk 'BEGIN {FS = "|" } NR==FNR{a[$6+0];next} $3+0 in a {print $0}' data1 data2? post also output of the command file data[12] Mar 22 at 11:01
  • Could you edit the question to show the result of printing the files with cat -vet? This shows visually any characters which are non-graphic. Mar 22 at 15:19

2 Answers 2

  1. You probably have DOS line endings, see why-does-my-tool-output-overwrite-itself-and-how-do-i-fix-it, so remove that if present (I'm removing that and any other trailing spaces with the sub() in my script below).
  2. If you're getting a blank line output then you have blank lines in each of your input files but I'd bet you aren't REALLY getting a blank line output and instead you're getting the 1 line of output you should get for 105.15 but the CR at the end is overwriting back to the start of the line - pipe the output to | cat -v to see if that's true.
  3. Your input has blanks before and after the |s in some places so you should set FS to match - FS=" *[|] *"
  4. You don't need to write {print $0} as that's the default behavior

Try this:

awk 'BEGIN{FS=" *[|] *"} {sub(/[[:space:]]+$/,"")} NR==FNR{a[$6];next} $3 in a' data1 data2
  • Your solution worked thanks !!! I removed your sub() script , it still worked so no DOS endings. What I don't understand is why awk needs to be told about anything between the field separators in this instance and not others. For example awk 'BEGIN {FS="|"} 1 ' ~/awk_tests/data1 produces the entire file printed out with no problems. Piping the original command tocat -v produced one blank line . Is this expected ? Just trying to get to the bottom of the problem to improve my understanding of awk .
    – ajr_chm
    Mar 22 at 23:14
  • Your values in data1 are not just |-separated. Consider |57.78 | - the value you want is 57.78 but that's not what's between the |s, it's 57.78<blank>. To get only 57.78 you need to tell awk to include blanks as part of the separator, hence FS=" *[|] *". Yes, awk 'BEGIN {FS="|"} 1 ' prints the whole file as you're not accessing any fields and so not using the FS that you're setting. A blank line of output just means you had blank lines in the input,
    – Ed Morton
    Mar 23 at 14:28

Your code works as-is for me, both with GNU awk 5.1.0 and with macOS awk 20200816.

Which version of awk are you using?

Note that you can also set the field separator with the -F command-line parameter; if you do that, then the BEGIN block is unnecessary.

  • I should have put that in the question. I used the -F command line parameter . Same result, an empty line .
    – ajr_chm
    Mar 22 at 10:22
  • Which version of awk? Mar 22 at 10:23
  • I am using gawk
    – ajr_chm
    Mar 22 at 10:25
  • That's still not a version number, but never mind. Is the data2 file perhaps in DOS/Windows mode? gawk doesn't seem to like that (I get no output if I convert it to CRLF line endings, though it doesn't seem to be affected by such changes in data1) Mar 22 at 10:30
  • GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.2.0) File is not in DOS/Windows format . Created in Gvim on Ubuntu 20.04
    – ajr_chm
    Mar 22 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.