39
 July 2022      mac os Monterey V12.1 
   awk --version 20200816
   GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21)

Why does awk -F work for most letters, but NOT for the letter t? I have the solution, but I would like to understand why awk fails for the letter t.

# Count 'e's
% echo "tweeter" | awk -F "e" '{print NF-1}'
3

# Count 'r's
% echo "tweeter" | awk -F "r" '{print NF-1}'
1

# (Attempt to) count 't's
% echo "tweeter" | awk -F "t" '{print NF-1}'
0   <=== ????

# Use gsub()
% echo "tweeter" | awk '{print gsub(/t/, "")}'
2
0

2 Answers 2

59

Because:

Normally, any number of blanks separate fields. In order to set the field separator to a single blank, use the -F option with a value of [ ]. If a field separator of t is specified, awk treats it as if \t had been specified and uses <TAB> as the field separator. In order to use a literal t as the field separator, use the -F option with a value of [t].

That's from the FreeBSD awk man page, and the utilities that come with macOS are usually some old FreeBSD versions or such.

$ printf 'foo\tbar\n' | awk -F t '{print NF-1}'
1
$ echo total | awk -F '[t]' '{print NF-1}'
2

In a way, that seems like a useful shorthand for files with tab-separated values, but what with other letters taken as-is, it's confusing. It only works like that with -F, using -v FS=t doesn't do it.

The feature is non-POSIX, as POSIX says that -F x is the same as -v FS=x. Most other awks I tested treated t as the the literal letter (some versions of gawk, mawk and Busybox).

The version of awk that e.g. Debian has in the original-awk package ("One True AWK" or "BWK awk" presumably from Brian W. Kernighan's initials) does support it, though, and at least Wikipedia seems to indicate that would be the same software FreeBSD uses. This one appears to be based on the version described in the 1988 book "The AWK Programming Language", but I'm not an expert on awk lineages and don't know if it has evolved significantly since then. That one is on github, but the documentation there doesn't seem to describe the feature. The special case can be seen in the code (where it's described as "a wart" in a comment).

You can get the same behaviour with GNU awk in BWK-awk compatibility mode, though.:

As a special case, in compatibility mode (see section Command-Line Options), if the argument to -F is ‘t’, then FS is set to the TAB character. If you type ‘-F\t’ at the shell, without any quotes, the ‘\’ gets deleted, so awk figures that you really want your fields to be separated with TABs and not ‘t’s.

8
  • 2
    I wonder if Mr Kernighan was accommodating users who didn't quote the value: awk -F \t ... and if sh and csh handled that differently. Jul 6 at 0:47
  • 3
    @glennjackman, the GNU documentation that Ed linked to there actually mentions just that. Both sh and csh drop the backslash there.
    – ilkkachu
    Jul 6 at 8:39
  • 1
    @QuartzCristal and Ed, it might not even matter, at least Wikipedia says the awk that FreeBSD uses is BWK awk / nawk. The version messages look the same between the macOS one and the Debian "original-awk" one. (And I'm not going to dig deeper. The one I have that came with macOS is an older version though)
    – ilkkachu
    Jul 6 at 8:57
  • 2
    FWIW, in rc -F\t remains as is as backslash is not a quoting operator there. In es or fish, -F\t becomes -F<TAB>. Jul 6 at 12:02
  • 2
    The interpretation of "t" (it happens to be quoted, as it should, in the OP) as TAB seems like a misfeature of BSD's awk. Jul 6 at 18:57
-4

the absolutely safest way, regardless of shell/env combo, is to use a square-cage and double-backslash for the tab

FS='[t]'         # for just lowercase "t"
FS='[\\411]'     # for the tab "\t"

with the doubling of \\ even though it's inside single quotes, shall any environment get too smart and try stripping one of them off, awk will still it as the underlying byte of tab itself.

if youre really paranoid about it, then do

FS='[\534\564]'  # for the tab "\t"
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.