9

I tried both

... | sort -k'1,1r' -k'2,2' -t'±'

... | sort -k'1,1r' -k'2,2' -t$'\xC2\xB1'


*$> sort: ±: Invalid argument*

I hope it is not unsupported by sort, but just that I am missing some escaping or special character handling.

My settings are:

localhost:~ user$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

localhost:~ user$ sort --version
2.3-Apple (99)
  • 1
    Can you use -t$'\xc2' as a workaround? – Freddy May 3 at 1:37
  • 1
    You would need to mention your character encoding and what type of platform you are using. I cannot repeat your problem on a certified UNIX. – schily May 3 at 5:07
  • GNU bash, version 3.2.57(1)-release , Darwin Kernel Version 18.7.0. I though all shells use utf-8 by default, do they? – Whimusical May 3 at 8:45
  • @Freddy no luck but thanks anyway! – Whimusical May 3 at 8:47
  • Can you paste the output of locale and sort --version into your question? – Mark Plotnick May 3 at 9:11
26

If you look at the source code of Apple macOS sort, it's little modified from its FreeBSD origin.

In particular, you find the same awkward/broken handling of the -t option as in FreeBSD.

            case 't':
                while (strlen(optarg) > 1) {
                    if (optarg[0] != '\\') {
                        errc(2, EINVAL, "%s", optarg);
                    }
                    optarg += 1;
                    if (*optarg == '0') {
                        *optarg = 0;
                        break;
                    }
                }

As you can see, the argument to -t is accepted only if it's empty (in which case NUL is the delimiter) or contains a single byte or starts with any number of \ characters followed by either a single byte (in which case that byte is taken as the delimiter), or 0 followed by anything, in which case the delimiter is NUL.

Examples:

  • -t '' or -t '\0', or -t '\\\\\0' or -t '\\0whatever delimits on the NUL character
  • -t '\t', -t '\\\t' delimits on t
  • -t '\', -t '\\\\\' delimits on backslash.

In any case, the delimiter can only be a single byte, that arcane extra processing is probably only there so -t '\0' can be used to specify a NUL delimiter for compatibility with GNU sort (FreeBSD's sort used to be GNU sort), or possibly (since that commit which was not even about the -t option) so that -t '\\' could also be used to specify \ as the delimiter (not something that GNU sort accepts).

So you can't use a multi-byte character as the delimiter.

Not many sort implementations allow a multi-byte character there. GNU or busybox sort don't either. ast-open's sort does though.

Here, you could swap the ± with a single-byte character (preferably one that is unlikely to occur in the input so it doesn't affect sorting) before sorting and restore afterwards. Thankfully FreeBSD's tr, and presumably macOS' tr as well supports multi-byte characters (contrary to GNU tr):

<input tr '±\1' '\1±' | sort -t $'\1' ... | tr '±\1' '\1±'
| improve this answer | |
  • I found a simplest solution, although this answer is so complete I am not going to add my answer. LC_ALL=UTF-8 sort -k'1,1r' -k'2,2' -t$'\xC2' – Whimusical May 3 at 19:43
  • 9
    It is truly sad that even in the year 2020, fundamental text processing tools are built on fundamentally broken assumptions that haven't been true in 3-5 decades, if ever. – Jörg W Mittag May 3 at 20:20
  • 4
    @Whimusical, I wouldn't rely on that. 0xC2 is the first byte of characters U+0080 to U+00BF, not just ±. That also means your second key will start with \xB1 which means invalid text. Some strcoll() implementations will choke on that. It's also possible it would break the day macOS fix their sort to support multibyte delimiters. – Stéphane Chazelas May 3 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.