7

I want to sort two files but I cannot get consistent results. It seems there are problems with collation but I cannot understand the reason. In sample files separator is a single space:

file1:

a
b
B
A

file2:

a 1
b 0
B 1
A 0

I use sort -k1,1 to sort these files and the output is:

sorted1:

a
A
b
B

sorted2:

A 0
a 1
b 0
B 1

I need those sorted files in a join and its currently complaining that the one of files is not sorted.

In my environment LC_COLLATE and LC_ALL are not set, LANG is set to en_US.UTF-8

With LC_ALL=C sort -k1,1 the output is:

sorted11:

A
B
a
b

sorted22:

A 0
B 1
a 1
b 0

I don't need a specific ordering, I just want it to be able to join the results. This way join works. To be safe I can also prepend join with LC_ALL=C.

My question

Why in sorted1 a is before A and in sorted2 a is after A? Whatever the collation is, it is for both sort commands and I am sorting based on column 1 that is identical in both input files.

Added output of ltrace -e strcoll

file1

sort->strcoll("B","A") =1
sort->strcoll("a","b") =-1 
sort->strcoll("a","A") =-7
a
sort->strcoll("b","A") =1
A
sort->strcoll("b","B") =-7
b
B
+++ exited (status 0) +++

file2

sort->strcoll("B 1","A 0") =1
sort->strcoll("a 1","b 0") =-1 
sort->strcoll("a 1","A 0") =1
A 0
sort->strcoll("a 1","B 1) =-1
a 1
sort->strcoll("b 0","B 1") =-1
b 0
B 1
+++ exited (status 0) +++
  • Can't reproduce this, what version of sort are you using ? – 123 Aug 19 '15 at 10:15
  • @User112638726, I am using sort 8.22 on centos 7 – Ashkan Aug 19 '15 at 10:19
  • Check that you don't have some unprintable characters in the file, like \0. For example with recode us..dump file. – jimmij Aug 19 '15 at 10:23
  • 1
    You probably want to use LC_ALL=C anyway for join, especially if the file contains non-ASCII characters as all UTF-8 glibc locales are buggy in that many different characters and collating elements sort the same there. – Stéphane Chazelas Aug 19 '15 at 11:31
  • 4
    I can reproduce with the sort in rpm.pbone.net/index.php3/stat/4/idpl/26640824/dir/centos_7/com/…, which seems completely broken in that regard, not with the base 8.22 from gnu.org. I can see the CentOS one has a large internationalisation patch which is probably the culprit. You may want to report it as a bug to CentOS if not already known. – Stéphane Chazelas Aug 19 '15 at 12:49
2

As Stéphane Chazelas said in the comment, it is a bug in the specific implementation of coreutils (in coreutils-8.22-11.el7) by CentOS/Red Hat, more specifically in the buggy internationalisation patch (coreutils-i18n.patch) they wrote and applied on top of GNU's coreutils-8.22.

I reported it here to CentOS and also here to Red Hat. It was already known at Red Hat and fixed there in coreutils-8.22-13.el7.

That one is not available yet for CentOS at this time (2015-08-20).

For completeness, note that the bug was also (incorrectly as the bug was not there) reported upstreams (at GNU's) where you'll find some more information about it.

0

Your default collation(en_US.UTF-8) cause to this. You should set LC_COLLATE value to order the text as directed.

 LC_COLLATE='C' sort -k1 file1
  • 1
    My question is why my default collation works this way? One of these can be true a is before A or a is after A not both. How can a collation result in both? – Ashkan Aug 19 '15 at 11:11
  • Not sort -k1, sort -k1 is the same as sort. It sorts on the section of the line that starts at the first field, so basically the whole line. sort -k1,1 to sort on the first field. – Stéphane Chazelas Aug 19 '15 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.