1

I'm experimenting with GNU sort and LC_COLLATE="en_US.UTF-8". I have a file called 'test':

1,0 1
10 2
1,0 3
10 4

With sort -k1,2 as well as with simple sort test the order doesn't change:

$ sort -k1,2 test
1,0 1
10 2
1,0 3
10 4

So, sort thinks that '1,0' is equal to '10' probably due to some quirks of LC_COLLATE (skipping punctuation?)

Now, when I use sort -k1,1 -k2,2, it gives me a different order:

$ sort -k1,1 -k2,2 test
10 2
10 4
1,0 1
1,0 3

and suddenly sort doesn't think that '10' is the same as '1,0' anymore.

What happened? Why isn't sort -k1,1 -k2,2 equivalent to sort -k1,2 in this case? Should it really be equivalent? Or have I misinterpreted the man page? (I tried versions 8.22 and 8.29 of coreutils, both have this behavior)

2

-k1,2 means “sort all lines, comparing the contents of all fields from 1 to 2 simultaneously”; so “1,0 1” is compared with “10 2” etc.

-k1,1 -k2,2 means “sort all lines, comparing the contents of field 1, and when two lines have the same content in field 1, comparing the contents of field 2”; so “1,0” is compared with “10”, then “2” with “4” etc.

What happens then, in both cases, boils down to collation, in particular weighting. Digits typically have a higher weight than punctuation and spacing. When comparing “1,0 1” and “10 2”, the difference due to the comma is ignored because the digits are different. When comparing “1,0” and “10”, the only difference is the comma, so it’s no longer ignored. See ISO 14651 for details.

You can set LC_COLLATE=C to get collation based only on character values, with no weights. Your examples both result in

1,0 1
1,0 3
10 2
10 4

when the “C” locale is used.

  • Thank you for the explanation. That's some elaborate collation rules! Good to know. I actually use LC_COLLATE=C locally, but when running jobs on a hadoop cluster, I don't have the same control over remote environments. But that's for a different question :) – lutyj Jul 19 '18 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.