10

The GNU sort command is not sorting words of different lengths with common prefixes correctly for me, but only when using a field delimiter to sort on one of multiple fields.

Here is the correct, expected sort behavior without using field delimiters:

$ cat /tmp/test0
b
c
ant
a
bcd
bc
cn

$ sort /tmp/test0
a
ant
b
bc
bcd
c
cn

Note that, for all words with a common string prefix, the shorter word sorts before the longer word. E.g. a is before ant, b is before bc is before bcd, etc. This is the accepted, standard way that English strings are sorted, e.g. in a dictionary.

However, this sorting behavior changes when you are attempting to sort tabular data (such as a CSV file), and sorting on one of the columns. Here's what that looks like:

$ cat /tmp/test1
b,foo
c,bar
ant,baz
a,foo
bcd,ty
bc,pe
cn,cn

$ sort /tmp/test1 -t, -k1
a,foo
ant,baz
bcd,ty
bc,pe
b,foo
c,bar
cn,cn

Note that the words with a common prefix of a and c are still being handled correctly, but strings with a common prefix of b are not; bcd sorts before bc sorts before b, all of which is incorrect! This behavior is stable; you always get the same result. I'm experiencing this exact same issue on a much larger CSV file and the sorting errors there are deterministically random, if that makes sense.

I've tried various flags to sort and none work to correct this behavior. -d and -s don't work. This is on GNU coreutils 9.4 sort for what it's worth.

So, is this just a bug with the sort command? Am I somehow using it incorrectly? Is there anything better I can do that will dictionary sort the CVS by words in the first column?

2
  • 2
    sort sorts as per the locale's collation order, and in most locales, that's similarly to what you'll find in a dictionary, where whitespace, punctuation are ignore in a first pass (have IGNORE as their primary weight). Try in the C locale if you expect an order based on byte value (LC_ALL=C sort...) or use sort -t, -k1,1 -k2,2 if you want to sort based on the first comma-separated field as the first key and the second one as the second key. -k1 (to sort on the portion of the line starting with the first field so the whole line) is the default so pointless. Commented May 24 at 16:38
  • 2
    To sort CSVs, have a look at the mlr (miller) utility. sort can only sort the most simple CSVs (without header, without quoting, with no newline in cells). Commented May 24 at 16:40

3 Answers 3

16

It's the way your current locale defines collations/sorting rules that's causing it, and how -kN uses field N to the end of the line when comparing lines, not just field N (And some locales will sort bc,pe before b,foo if they ignore the commas).

Use -k1,1 to only use that specific field, or specify the "C" locale and you should get the expected results:

$ LC_ALL=en_US.utf8 sort -t, -k1 test.txt
a,foo
ant,baz
bcd,ty
bc,pe
b,foo
c,bar
cn,cn

$ LC_ALL=en_US.utf8 sort -t, -k1,1 test.txt
a,foo
ant,baz
b,foo
bc,pe
bcd,ty
c,bar
cn,cn

$ LC_ALL=C sort -t, -k1 test.txt
a,foo
ant,baz
b,foo
bc,pe
bcd,ty
c,bar
cn,cn
4
  • Thank you for that, I can confirm it works with LC_ALL=C. However, the part I still don't understand (and what confused me) is why the sorting order over the same set of strings is different depending on whether one is a column in tabular data. If the bare non-tabular file had sorted as bcd, bc, b as well this would have been a lot simpler to figure out. It's the inconsistency that's strange, and I can see potentially tripping up a lot of people. Commented May 24 at 16:58
  • 4
    It's because , is sorted after c with LC_ALL=en_US.utf8 and before with LC_ALL=C.
    – ctx
    Commented May 24 at 17:09
  • 4
    @BenMcIlwain I keep forgetting that -k N uses field N to the end of the line. Try -k 1,1 instead and you also get the expected sort order.
    – Shawn
    Commented May 24 at 17:13
  • 8
    Oh my god, that's the real answer then. -k 1,1. Yeeesh! LC_ALL=C only works by accident because of the selected delimiter's byte value, but it wouldn't work with other delimiters; -k 1,1 would work with all. Commented May 24 at 17:25
14

The answer turns out to be that, despite some example usages online to the contrary, the -k flag takes TWO parameters, so it needs to be written as -k 1,1. Otherwise, with -k 1, it's not getting a stop field number, and thus just going through the entire line. So the anomalous sort behavior is actually caused by the UTF-8 representation of the , delimiter falling somewhere in-between the other ASCII characters.

Thanks to Stéphane Chazelas's comments above.

4

According to the comment of Stéphane Chazelas you are sorting the wohle line and not only the first field. With LC_ALL=C , sorts before c:

% LC_ALL=en_US.utf8 sort -t, -k1,1  test --debug 
sort: text ordering performed using ‘en_US.utf8’ sorting rules
a,foo
_
_____
ant,baz
___
_______
b,foo
_
_____
bc,pe
__
_____
bcd,ty
___
______
c,bar
_
_____
cn,cn
__
_____
% LC_ALL=en_US.utf8 sort -t, -k1  test --debug 
sort: text ordering performed using ‘en_US.utf8’ sorting rules
a,foo
_____
_____
ant,baz
_______
_______
bcd,ty
______
______
bc,pe
_____
_____
b,foo
_____
_____
c,bar
_____
_____
cn,cn
_____
_____

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .