awk when both delimiter and quotes are used for a field

Question

I have a file in the following format:

field1|field2|field3
field1|"field2|field2"|field3

Notice the second row contains double quotes. The string within the double quotes belongs to field 2. How do extract this using awk? I've been googling with no results. I tried this with no luck as well

FS='"| "|^"|"$' '{print $2}'  

Answer 1

If you have a recent version of gawk you're in luck. There's the FPAT feature, documented here

awk 'BEGIN {
 FPAT = "([^|]+)|(\"[^\"]+\")"
}
{
 print "NF = ", NF
 for (i = 1; i <= NF; i++) {
    sub(/"$/, "", $i); sub(/^"/, "", $i);printf("$%d = %s\n", i, $i)
 }
}' file

NF =  3
$1 = field1
$2 = field2
$3 = field3
NF =  3
$1 = field1
$2 = field2|field2
$3 = field3

Answer 2

This is something that you get in csv - if the delimiter is part of the field, it gets quoted. That suddenly makes the task of parsing it MUCH harder, because you can't just split on a delim.

Fortunately, if perl is an option, you have the Text::CSV module that handles this case:

#!/usr/bin/env perl
use strict;
use warnings;

use Text::CSV;

my $csv = Text::CSV -> new ( { 'sep_char' => '|' } );

while ( my $row =  $csv -> getline ( *STDIN ) ) {
   print $row -> [1],"\n";
}

Could probably condense this to an inline/pipeable if you prefer - something like:

perl -MText::CSV -e 'print map { $_ -> [1] ."\n" } @{ Text::CSV -> new ( { 'sep_char' => '|' } ) -> getline_all ( *ARGV )};

Answer 3

You may want to format this data with sed so it can be parsed by awk more easily. for example:

$ sed 's/"//g' awktest1.txt 
field1|field2|field3
field1|field2|field2|field3

$ sed 's/"//g' awktest1.txt > awktest2.txt

$ awk 'BEGIN {FS = "|"} ; {print $2}' awktest2.txt 
field2
field2

But then again, I don't know the nature of the data you are working with.