9

What is the difference between gawk and mawk with regards to field separators? In particular, I'm trying to figure out what is going on here:

mawk:

$ echo "100+50°20.5" | mawk -F '[+°.]' '{ print $1" - "$2" - "$3" - "$4" - "$5; }'
100 - 50 -  - 20 - 5

gawk:

$ echo "100+50°20.5" | gawk -F '[+°.]' '{ print $1" - "$2" - "$3" - "$4" - "$5; }'
100 - 50 - 20 - 5 -

It looks like mawk is introducing an extra field somehow. What is the correct behavior?

4
  • 2
    see the -b (--characters-as-bytes) switch of the GNU awk or pass LC_ALL=C gawk or LC_ALL=C mawk for the same result; see also Locales- Where You Are Makes A Difference Mar 20, 2023 at 6:16
  • 6
    The important (and not necessarily obvious) thing here is ° is a multi-byte character. Mar 20, 2023 at 6:30
  • 2
  • 1
    @KamilMaciorowski Or at least, ° is multi-byte in UTF-8. (I remember the days when people could be using all sorts of character encodings, and you couldn't make any assumptions at all! Things are much better now that UTF-8 is the default, but it's not yet universal.)
    – gidds
    Mar 21, 2023 at 22:27

2 Answers 2

21

To understand what’s happening here, look at the bytes provided as input to AWK:

$ od -t x1 <<<"100+50°20.5"
0000000 31 30 30 2b 35 30 c2 b0 32 30 2e 35 0a
0000015

In UTF-8, “°” is a multi-byte character, represented by 0xC2 0xB0. MAWK doesn’t support multi-byte characters, so it treats the regular expression provided as a field separator as matching one of the four bytes 0x2B (“+”), 0xC2 (“°” part one), 0xB0 (“°” part two), and 0x2E (“.”). These match four times in the input string, resulting in five fields:

  • “100”
  • “50”
  • the empty string (between 0xC2 and 0xB0)
  • “20”
  • “5”

GAWK takes the current locale into account and supports multi-byte characters by default, so it matches “°” as a character and finds four fields. This can be disabled by using the -b option or by switching to a non-multi-byte locale with LC_ALL=C.

15

As wisely explained in StephenKitt and in the comments, ° multibyte char is being split in [°] context...

One way to get around this is to use the "or" regex operator:

awk -F '[+.]|°' ...
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .