0

Why does adding identical characters to the end of filenames change the sort order as shown below? In any sane string comparison method I would expect the comparison of 2 strings to not differ if the same string is added to the end of both of them.

$ type ls
ls is aliased to `ls --color=auto`
$ touch I II III IV V VI
$ ls
I  II  III  IV  V  VI
$ rm -f *
$ for n in I II III IV V VI; do touch "$n x"; done
$ ls 
III x  II x  IV x  I x  VI x  V x

Mean while Python works like I would expect:

>>> ns = ['I', 'II', 'III', 'IV', 'V', 'VI']
>>> sorted(ns)
['I', 'II', 'III', 'IV', 'V', 'VI']
>>> sorted(n + ' x' for n in ns)
['I x', 'II x', 'III x', 'IV x', 'V x', 'VI x']
>>> 
  • 1
    Hint: LANG=C ls and LANG=en_US.UTF-8 ls – Cyrus Dec 26 '15 at 21:58
  • “I would expect the comparison of 2 strings to not differ if the same string is added to the end of both of them.” — No, that is not true of any sane string comparison. It fails in particular when one of the string is a prefix of another: a < aa but ab > aab. – Gilles 'SO- stop being evil' Dec 26 '15 at 22:57
7

as the comment from cyrus hinted, the reason is the collating rules (=locale-aware string comparison). In your case, with most non C/POSIX locales (e.g. 'en_US.UTF-8'), space characters are ignored when comparing strings, thus "Ix" and "I x" are evaluated to be equal, and as a result, "II" comes before "Ix".

See the following:

$ touch I "I x" "Ix" "II" IIx "II x"

When using a non-C locale, often spaces are ignored when comparing strings:

$ locale | grep LANG
LANG=en_US.UTF-8
LANGUAGE=en_US

$ ls -lhog
total 0
-rw-rw-r-- 1 0 Dec 26 17:12 I
-rw-rw-r-- 1 0 Dec 26 17:12 II
-rw-rw-r-- 1 0 Dec 26 17:12 IIx
-rw-rw-r-- 1 0 Dec 26 17:12 II x
-rw-rw-r-- 1 0 Dec 26 17:12 Ix
-rw-rw-r-- 1 0 Dec 26 17:12 I x

Forcing a POSIX/C locale, filenames are compared by ASCII characters, thus 'space' characters comes before 'x' or 'I':

$ LC_ALL=C ls -lhog
total 0
-rw-rw-r-- 1 0 Dec 26 17:12 I
-rw-rw-r-- 1 0 Dec 26 17:12 I x
-rw-rw-r-- 1 0 Dec 26 17:12 II
-rw-rw-r-- 1 0 Dec 26 17:12 II x
-rw-rw-r-- 1 0 Dec 26 17:12 IIx
-rw-rw-r-- 1 0 Dec 26 17:12 Ix

To see the actual comparison function results (strcoll(3)), use ltrace like so:

$ LC_ALL=C ltrace -e strcoll ls -lhog
ls->strcoll("I", "II x")       = -73
ls->strcoll("Ix", "I")         = 120
ls->strcoll("Ix", "II x")      = 47
ls->strcoll("I x", "II")       = -41
ls->strcoll("IIx", "I x")      = 41
ls->strcoll("IIx", "II")       = 120
ls->strcoll("I x", "I")        = 32
ls->strcoll("I x", "II x")     = -41
ls->strcoll("II", "II x")      = -32
ls->strcoll("IIx", "II x")     = 88
ls->strcoll("IIx", "Ix")       = -47
total 0
-rw-rw-r-- 1 0 Dec 26 17:12 I
-rw-rw-r-- 1 0 Dec 26 17:12 I x
-rw-rw-r-- 1 0 Dec 26 17:12 II
-rw-rw-r-- 1 0 Dec 26 17:12 II x
-rw-rw-r-- 1 0 Dec 26 17:12 IIx
-rw-rw-r-- 1 0 Dec 26 17:12 Ix
+++ exited (status 0) +++

$ LC_ALL=en_US.UTF-8 ltrace -e strcoll ls -lhog
ls->strcoll("I", "II x")       = -1
ls->strcoll("Ix", "I")         = 1
ls->strcoll("Ix", "II x")      = 15
ls->strcoll("I x", "II")       = 15
ls->strcoll("IIx", "II")       = 1
ls->strcoll("IIx", "I x")      = -15
ls->strcoll("II", "I")         = 1
ls->strcoll("II", "II x")      = -1
ls->strcoll("IIx", "II x")     = -1
ls->strcoll("I x", "II x")     = 15
ls->strcoll("I x", "Ix")       = 1
total 0
-rw-rw-r-- 1 0 Dec 26 17:12 I
-rw-rw-r-- 1 0 Dec 26 17:12 II
-rw-rw-r-- 1 0 Dec 26 17:12 IIx
-rw-rw-r-- 1 0 Dec 26 17:12 II x
-rw-rw-r-- 1 0 Dec 26 17:12 Ix
-rw-rw-r-- 1 0 Dec 26 17:12 I x
+++ exited (status 0) +++

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.