Using NF with FPAT – what am I doing wrong?

Question

When NF is used with FPAT regex, the comma is considered a field. I prefer using NF and FPAT:

1) NF – to limit the output to the actual number of fields for the record

2) FPAT – to handle an embedded comma in a quoted field like line 3:

 "Bus Driver, City/Transit",51

3) the awk script is used for several input files that have a different number of columns for record 6 - record 6 is the column name/header for the contents of the file...

The output from testing, the first, test1, uses a fixed value for number of fields, the second, test2, uses NF for the number of fields.

using gawk 4.1.4

    BEGIN {
            FPAT = "(^,)|([^,]+)|(\"[^\"]+\")"
            OFS = "\t"
        }

    NR == 6 {

       for (i = 1; 6 >= i; ++i) {
       #for (i = 1; NF >= i; ++i) {

           colName[i] = $i
           print "Column Name: " colName[i]

    }    

      { print "", "number of fields: " NF }
    }

Input File starting at record 6: NR == 6 {...

    Occupation,States Licensed 
    Barber,51 
    "Bus Driver, City/Transit",51

The output I expect/want:

    Column Name: Occupation
    Column Name: States Licensed
        number of fields: 2

test 1: for (i = 1; 6 >= i; ++i) {...

output is correct - what I expect/want, except, of course, for the 4 columns/fields that are not valid but are shown because of using a fixed value of 6.

    Column Name: Occupation
    Column Name: States Licensed
    Column Name: 
    Column Name: 
    Column Name: 
    Column Name: 
        number of fields: 2

test 2: for (i = 1; NF >= i; ++i) {...

output is NOT what I expect/want; note the comma is a indicate as a field

    Column Name: Occupation
    Column Name: ,
    Column Name: States Licensed
        number of fields: 3

Answer 1

0. Congratulations.  You seem to have found a bug in gawk.

I’ve reduced this to a very small example.  (It might be possible to demonstrate the glitch with a simpler FPAT string, but I didn’t feel like spending another ten minutes on that.)  Basically, for input like foo,bar, we can get two different results.

Case A:

NF = 2
$1 = foo
$2 = bar
$3 =

and

Case B:

NF = 3
$1 = foo
$2 = ,
$3 = bar

This code produces Case B:

BEGIN {
        FPAT = "^,|[^,]+"
}

{
        print "NF =", NF
        print "$1 =", $1; print "$2 =", $2; print "$3 =", $3
}

(I removed the parentheses from FPAT, because they aren’t needed; I removed the part of the regular expression that handles quoted strings maybe containing comma(s), and I cut the code down to a bare minimum.)

Use

echo foo,bar | awk -f name_of_the_above_awk_script

But — in gawk version 4.1.1, at least — if I access $1 before we access NF, then I get Case A.  You can demonstrate this by switching the order of the print statements, or by this ridiculous kluge:

{
        temp = $1                       # We will never use this.
        print "NF =", NF
        print "$1 =", $1; print "$2 =", $2; print "$3 =", $3
}

This is clearly a bug; there’s no way that accessing a field should change the values of other things. 

1. So we have a work-around.

Just add temp = $1 before your for loop, and I expect you’ll get the result you want (using NF).

2. The real (?) answer:

In the above, I deliberately avoided referring to either Case A or Case B as “right” or “wrong”.  Case A is the one you want, but Case B might actually be the correct result for the value of FPAT that you’re using.  It seems to be saying that you want a field to be

• a string beginning with a comma, or
• a string of one or more characters that aren’t comma, or
• a quote, a string of one or more characters that aren’t quote, and another quote.

But you don’t want a comma to be a field; you just want the second and third options.  I find that setting

FPAT = "[^,]+|\"[^\"]+\""

will give you the correct results.