1

When NF is used with FPAT regex, the comma is considered a field. I prefer using NF and FPAT:

1) NF – to limit the output to the actual number of fields for the record

2) FPAT – to handle an embedded comma in a quoted field like line 3:

 "Bus Driver, City/Transit",51

3) the awk script is used for several input files that have a different number of columns for record 6 - record 6 is the column name/header for the contents of the file...

The output from testing, the first, test1, uses a fixed value for number of fields, the second, test2, uses NF for the number of fields.

using gawk 4.1.4

    BEGIN {
            FPAT = "(^,)|([^,]+)|(\"[^\"]+\")"
            OFS = "\t"
        }

    NR == 6 {

       for (i = 1; 6 >= i; ++i) {
       #for (i = 1; NF >= i; ++i) {

           colName[i] = $i
           print "Column Name: " colName[i]

    }    

      { print "", "number of fields: " NF }
    }

Input File starting at record 6: NR == 6 {...

    Occupation,States Licensed 
    Barber,51 
    "Bus Driver, City/Transit",51

The output I expect/want:

    Column Name: Occupation
    Column Name: States Licensed
        number of fields: 2

test 1: for (i = 1; 6 >= i; ++i) {...

output is correct - what I expect/want, except, of course, for the 4 columns/fields that are not valid but are shown because of using a fixed value of 6.

    Column Name: Occupation
    Column Name: States Licensed
    Column Name: 
    Column Name: 
    Column Name: 
    Column Name: 
        number of fields: 2

test 2: for (i = 1; NF >= i; ++i) {...

output is NOT what I expect/want; note the comma is a indicate as a field

    Column Name: Occupation
    Column Name: ,
    Column Name: States Licensed
        number of fields: 3
  • The problem is your regex I think - try FPAT = "\"[^\"]*\"|[^\",]*" (a possibly empty sequence of non-quotes surrounded by quotes, or a possibly empty sequence of not-comma-or-quotes). Or more readably gawk -v FPAT='"[^"]*"|[^",]*' '<stuff>' – steeldriver Mar 2 at 1:25
1

0. Congratulations.  You seem to have found a bug in gawk.

I’ve reduced this to a very small example.  (It might be possible to demonstrate the glitch with a simpler FPAT string, but I didn’t feel like spending another ten minutes on that.)  Basically, for input like foo,bar, we can get two different results.

Case A:

NF = 2
$1 = foo
$2 = bar
$3 =

and

Case B:

NF = 3
$1 = foo
$2 = ,
$3 = bar

This code produces Case B:

BEGIN {
        FPAT = "^,|[^,]+"
}

{
        print "NF =", NF
        print "$1 =", $1; print "$2 =", $2; print "$3 =", $3
}

(I removed the parentheses from FPAT, because they aren’t needed; I removed the part of the regular expression that handles quoted strings maybe containing comma(s), and I cut the code down to a bare minimum.)

Use

echo foo,bar | awk -f name_of_the_above_awk_script

But — in gawk version 4.1.1, at least — if I access $1 before we access NF, then I get Case A.  You can demonstrate this by switching the order of the print statements, or by this ridiculous kluge:

{
        temp = $1                       # We will never use this.
        print "NF =", NF
        print "$1 =", $1; print "$2 =", $2; print "$3 =", $3
}

This is clearly a bug; there’s no way that accessing a field should change the values of other things. 

1. So we have a work-around.

Just add temp = $1 before your for loop, and I expect you’ll get the result you want (using NF).

2. The real (?) answer:

In the above, I deliberately avoided referring to either Case A or Case B as “right” or “wrong”.  Case A is the one you want, but Case B might actually be the correct result for the value of FPAT that you’re using.  It seems to be saying that you want a field to be

  • a string beginning with a comma, or
  • a string of one or more characters that aren’t comma, or
  • a quote, a string of one or more characters that aren’t quote, and another quote.

But you don’t want a comma to be a field; you just want the second and third options.  I find that setting

FPAT = "[^,]+|\"[^\"]+\""

will give you the correct results.

  • Interesting and such a simple solution. The regex I used was copied from the GNU AWK website/page, "The GNU Awk User’s Guide": GNU.org except that I added the (^,)| at the start for some now unknown reason (doh!?). Thank you for your time, detailed solution and well laid-out presentation - much appreciated and I learned something. I wonder if I should let GNU.org know about the error in the manual, re: the parenthesis in the regex. Would it be OK if I quote your solution? – TomM Mar 2 at 16:47
  • Forgot to mention that I didn't have to add temp = $1. It works fine without it. – TomM Mar 2 at 16:55
  • (1) The inclusion of the parentheses in FPAT is (as far as I can tell) unnecessary, but it’s harmless; I wouldn’t call it an error.  Adding the (^,)| at the start was the error.  But, yes, you may quote me if you say my name and link to the answer. (2) Adding temp = $1 was a kludgy workaround to get the program to produce the desired results even with the wrong FPAT.  So you don’t need to use the kludgy workaround and also fix FPAT.  Sorry I wasn’t clearer about that. – G-Man Says 'Reinstate Monica' Mar 2 at 17:19
  • using gawk 4.2.1 under Debian Testing, i got case B with our without temp=$1 – George Vasiliou Mar 2 at 21:36
  • 1
    @TomM While you don't (yet) have enough reputation to upvote helpful answers, I suggest you accept this answer (click on the check mark next to it). It's highly unlikely that you'll get a better one. See unix.stackexchange.com/help/someone-answers BTW, I was the one who upvoted your question because it was such a good one. Welcome to Unix & Linux! – Anthony G - justice for Monica Mar 9 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.