What is the difference between gawk and mawk with regards to field separators?

Question

What is the difference between gawk and mawk with regards to field separators? In particular, I'm trying to figure out what is going on here:

mawk:

$ echo "100+50°20.5" | mawk -F '[+°.]' '{ print $1" - "$2" - "$3" - "$4" - "$5; }'
100 - 50 -  - 20 - 5

gawk:

$ echo "100+50°20.5" | gawk -F '[+°.]' '{ print $1" - "$2" - "$3" - "$4" - "$5; }'
100 - 50 - 20 - 5 -

It looks like mawk is introducing an extra field somehow. What is the correct behavior?

Answer 1

To understand what’s happening here, look at the bytes provided as input to AWK:

$ od -t x1 <<<"100+50°20.5"
0000000 31 30 30 2b 35 30 c2 b0 32 30 2e 35 0a
0000015

In UTF-8, “°” is a multi-byte character, represented by 0xC2 0xB0. MAWK doesn’t support multi-byte characters, so it treats the regular expression provided as a field separator as matching one of the four bytes 0x2B (“+”), 0xC2 (“°” part one), 0xB0 (“°” part two), and 0x2E (“.”). These match four times in the input string, resulting in five fields:

• “100”
• “50”
• the empty string (between 0xC2 and 0xB0)
• “20”
• “5”

GAWK takes the current locale into account and supports multi-byte characters by default, so it matches “°” as a character and finds four fields. This can be disabled by using the -b option or by switching to a non-multi-byte locale with LC_ALL=C.

Answer 2

As wisely explained in StephenKitt and in the comments, ° multibyte char is being split in [°] context...

One way to get around this is to use the "or" regex operator:

awk -F '[+.]|°' ...