Unix sort does not accept the unconventional delimiter '±' as a field separator

Question

I tried both

... | sort -k'1,1r' -k'2,2' -t'±'

... | sort -k'1,1r' -k'2,2' -t$'\xC2\xB1'


*$> sort: ±: Invalid argument*

I hope it is not unsupported by sort, but just that I am missing some escaping or special character handling.

My settings are:

localhost:~ user$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

localhost:~ user$ sort --version
2.3-Apple (99)

Answer 1

If you look at the source code of Apple macOS sort, it's little modified from its FreeBSD origin.

In particular, you find the same awkward/broken handling of the -t option as in FreeBSD.

            case 't':
                while (strlen(optarg) > 1) {
                    if (optarg[0] != '\\') {
                        errc(2, EINVAL, "%s", optarg);
                    }
                    optarg += 1;
                    if (*optarg == '0') {
                        *optarg = 0;
                        break;
                    }
                }

As you can see, the argument to -t is accepted only if it's empty (in which case NUL is the delimiter) or contains a single byte or starts with any number of \ characters followed by either a single byte (in which case that byte is taken as the delimiter), or 0 followed by anything, in which case the delimiter is NUL.

Examples:

• -t '' or -t '\0', or -t '\\\\\0' or -t '\\0whatever delimits on the NUL character
• -t '\t', -t '\\\t' delimits on t
• -t '\', -t '\\\\\' delimits on backslash.

In any case, the delimiter can only be a single byte, that arcane extra processing is probably only there so -t '\0' can be used to specify a NUL delimiter for compatibility with GNU sort (FreeBSD's sort used to be GNU sort), or possibly (since that commit which was not even about the -t option) so that -t '\\' could also be used to specify \ as the delimiter (not something that GNU sort accepts).

So you can't use a multi-byte character as the delimiter.

Not many sort implementations allow a multi-byte character there. GNU or busybox sort don't either. ast-open's sort does though.

Here, you could swap the ± with a single-byte character (preferably one that is unlikely to occur in the input so it doesn't affect sorting) before sorting and restore afterwards. Thankfully FreeBSD's tr, and presumably macOS' tr as well supports multi-byte characters (contrary to GNU tr):

<input tr '±\1' '\1±' | sort -t $'\1' ... | tr '±\1' '\1±'