Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

When sorting file names, ls ignores characters like -,_. I expected it to use those characters in sorting as well.

An example:

touch a1 a2 a-1 a-2 a_1 a_2 a.1 a.2 a,1 a,2

Now display these files with ls -1:

a1
a_1
a-1
a,1
a.1
a2
a_2
a-2
a,2
a.2

What I expected was something like this:

a1
a2
a,1
a,2
a.1
a.2
a_1
a_2
a-1
a-2

i.e. I expected the non-alphanumeric characters to be taken into account when sorting.

Can anyone explain this behaviour? Is this behaviour mandated by a standard? Or is this due the encoding being UTF-8?

Update: It seems that this is related to UTF-8 sorting:

$ LC_COLLATE=C ls -1
a,1
a,2
a-1
a-2
a.1
a.2
a1
a2
a_1
a_2
share|improve this question
2  
UTF-8 and ASCII are identical if all you're using is the first 128 codepoints (which your example is). What happens if you do LC_COLLATE=C ls? –  Alexios Apr 1 '12 at 9:56
    
The problem is not that ASCII and UTF-8 are identical, it's rather that UTF-8 has its own collation (sorting) rules. –  daniel kullmann Apr 1 '12 at 10:01
    
Yes, it's true that [_-,.] are being grouped and somehow semi-ignored. I don't know exactly how or where such collation is defined, but it must be a collation issue, because simply, and only, changing the collation to C (via LC_COLLATE=C ls -l) is enough to give you the sort order you expected (assuming the LC_ALL is not overriding LC_COLLATE). This holds true for the entire range of characters in the Unicode Basic Multilingual Plane... I've edited my answer to include an example script which bears this out... –  Peter.O Apr 1 '12 at 14:29
    
if you don't like how it works, you can create an alias and put it in your ~/.profile: alias ls='LC_COLLATE=C ls'</kbd> –  jippie Apr 1 '12 at 16:37
add comment

2 Answers

up vote 5 down vote accepted

This has nothing to do with the charset. Rather, it's the language that determines the collation order. The libc examines the language presented in $LC_COLLATE/$LC_ALL/$LANG and looks up its collation rules (e.g. /usr/share/i18n/locales/* for GLibC) and orders the text as directed.

share|improve this answer
add comment

EDIT: Added test for data sorted with LC_COLLATE=C


The default collate sequence is treating those "punctuation-type" characters as being of equal value.. Use LC_COLLATE=C to treat them in codepoint order ..

for i in 'a1' 'a_1' 'a-1' 'a,1' 'a.1' 'a2' 'a_2' 'a-2' 'a,2' 'a.2' ;do
  echo $i; 
done |LC_COLLATE=C sort

Output

a,1
a,2
a-1
a-2
a.1
a.2
a1
a2
a_1
a_2

The following code tests all valid UTF-8 chars in the Basic Multilingual Plane (except for \x00 and \x0a; for simplicity)
It compares a file in a known (generated) ascending sequence, against that file randomly sorted and then sorted again with LC_COLLATE=C. The result shows that the C sequence is identical to the original generated sequence.

{ i=0 j=0 k=0 l=0
  for i in {0..9} {A..F} ;do
  for j in {0..9} {A..F} ;do
  for k in {0..9} {A..F} ;do
  for l in {0..9} {A..F} ;do
     (( 16#$i$j$k$l == 16#0000 )) && { printf '.' >&2; continue; }
     (( 16#$i$j$k$l == 16#000A )) && { printf '.' >&2; continue; }
     (( 16#$i$j$k$l >= 16#D800    && 
        16#$i$j$k$l <= 16#DFFF )) && { printf '.' >&2; continue; }
     (( 16#$i$j$k$l >= 16#FFFE )) && { printf '.' >&2; continue; }
     echo 0x"$i$j$k$l" |recode UTF-16BE/x4..UTF-8 || { echo "ERROR at codepoint $i$j$k$l " >&2; continue; } 
     echo 
  done
  done
  done; echo -n "$i$j$k$l " >&2
  done; echo >&2
} >listGen

             sort -R listGen    > listRandom
LC_COLLATE=C sort    listRandom > listCsort 

diff <(cat listGen;   echo "last line of listOrig " ) \
     <(cat listCsort; echo "last line of listCsort" )
echo 
cmp listGen listCsort; echo 'cmp $?='$?

Output:

63485c63485
< last line of listOrig 
---
> last line of listCsort

cmp $?=0
share|improve this answer
2  
Where is that documented? Is that part of the Unicode standard? –  daniel kullmann Apr 1 '12 at 10:04
    
Actually, they don't get the same value; those characters are simply ignored when sorting. If they were treated as having an equal value, the sorting order a_1 a2 a_2 would be impossible. –  daniel kullmann Apr 1 '12 at 10:07
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.