Why does awk -F work for most letters, but not for the letter "t"?

Question

 July 2022      mac os Monterey V12.1 
   awk --version 20200816
   GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21)

Why does awk -F work for most letters, but NOT for the letter t? I have the solution, but I would like to understand why awk fails for the letter t.

# Count 'e's
% echo "tweeter" | awk -F "e" '{print NF-1}'
3

# Count 'r's
% echo "tweeter" | awk -F "r" '{print NF-1}'
1

# (Attempt to) count 't's
% echo "tweeter" | awk -F "t" '{print NF-1}'
0   <=== ????

# Use gsub()
% echo "tweeter" | awk '{print gsub(/t/, "")}'
2

Answer 1

Because:

Normally, any number of blanks separate fields. In order to set the field separator to a single blank, use the -F option with a value of [ ]. If a field separator of t is specified, awk treats it as if \t had been specified and uses <TAB> as the field separator. In order to use a literal t as the field separator, use the -F option with a value of [t].

That's from the FreeBSD awk man page, and the utilities that come with macOS are usually some old FreeBSD versions or such.

$ printf 'foo\tbar\n' | awk -F t '{print NF-1}'
1
$ echo total | awk -F '[t]' '{print NF-1}'
2

In a way, that seems like a useful shorthand for files with tab-separated values, but what with other letters taken as-is, it's confusing. It only works like that with -F, using -v FS=t doesn't do it.

The feature is non-POSIX, as POSIX says that -F x is the same as -v FS=x. Most other awks I tested treated t as the the literal letter (some versions of gawk, mawk and Busybox).

The version of awk that e.g. Debian has in the original-awk package ("One True AWK" or "BWK awk" presumably from Brian W. Kernighan's initials) does support it, though, and at least Wikipedia seems to indicate that would be the same software FreeBSD uses. This one appears to be based on the version described in the 1988 book "The AWK Programming Language", but I'm not an expert on awk lineages and don't know if it has evolved significantly since then. That one is on github, but the documentation there doesn't seem to describe the feature. The special case can be seen in the code (where it's described as "a wart" in a comment).

You can get the same behaviour with GNU awk in BWK-awk compatibility mode, though.:

As a special case, in compatibility mode (see section Command-Line Options), if the argument to -F is ‘t’, then FS is set to the TAB character. If you type ‘-F\t’ at the shell, without any quotes, the ‘\’ gets deleted, so awk figures that you really want your fields to be separated with TABs and not ‘t’s.

Answer 2

the absolutely safest way, regardless of shell/env combo, is to use a square-cage and double-backslash for the tab

FS='[t]'         # for just lowercase "t"

FS='[\\411]'     # for the tab "\t"

with the doubling of \\ even though it's inside single quotes, shall any environment get too smart and try stripping one of them off, awk will still it as the underlying byte of tab itself.

if youre really paranoid about it, then do

FS='[\534\564]'  # for the tab "\t"