9

I want to sort two files but I cannot get consistent results. It seems there are problems with collation but I cannot understand the reason. In sample files separator is a single space:

file1:

a
b
B
A

file2:

a 1
b 0
B 1
A 0

I use sort -k1,1 to sort these files and the output is:

sorted1:

a
A
b
B

sorted2:

A 0
a 1
b 0
B 1

I need those sorted files in a join and its currently complaining that the one of files is not sorted.

In my environment LC_COLLATE and LC_ALL are not set, LANG is set to en_US.UTF-8

With LC_ALL=C sort -k1,1 the output is:

sorted11:

A
B
a
b

sorted22:

A 0
B 1
a 1
b 0

I don't need a specific ordering, I just want it to be able to join the results. This way join works. To be safe I can also prepend join with LC_ALL=C.

My question

Why in sorted1 a is before A and in sorted2 a is after A? Whatever the collation is, it is for both sort commands and I am sorting based on column 1 that is identical in both input files.

Added output of ltrace -e strcoll

file1

sort->strcoll("B","A") =1
sort->strcoll("a","b") =-1 
sort->strcoll("a","A") =-7
a
sort->strcoll("b","A") =1
A
sort->strcoll("b","B") =-7
b
B
+++ exited (status 0) +++

file2

sort->strcoll("B 1","A 0") =1
sort->strcoll("a 1","b 0") =-1 
sort->strcoll("a 1","A 0") =1
A 0
sort->strcoll("a 1","B 1) =-1
a 1
sort->strcoll("b 0","B 1") =-1
b 0
B 1
+++ exited (status 0) +++
Improve this question
17
  • Can't reproduce this, what version of sort are you using ?
    – 123
    Aug 19, 2015 at 10:15
  • @User112638726, I am using sort 8.22 on centos 7
    – a5hk
    Aug 19, 2015 at 10:19
  • Check that you don't have some unprintable characters in the file, like \0. For example with recode us..dump file.
    – jimmij
    Aug 19, 2015 at 10:23
  • 2
    You probably want to use LC_ALL=C anyway for join, especially if the file contains non-ASCII characters as all UTF-8 glibc locales are buggy in that many different characters and collating elements sort the same there.
    – Stéphane Chazelas
    Aug 19, 2015 at 11:31
  • 5
    I can reproduce with the sort in rpm.pbone.net/index.php3/stat/4/idpl/26640824/dir/centos_7/com/…, which seems completely broken in that regard, not with the base 8.22 from gnu.org. I can see the CentOS one has a large internationalisation patch which is probably the culprit. You may want to report it as a bug to CentOS if not already known.
    – Stéphane Chazelas
    Aug 19, 2015 at 12:49

2 Answers 2

Reset to default
4

As Stéphane Chazelas said in the comment, it is a bug in the specific implementation of coreutils (in coreutils-8.22-11.el7) by CentOS/Red Hat, more specifically in the buggy internationalisation patch (coreutils-i18n.patch) they wrote and applied on top of GNU's coreutils-8.22.

I reported it here to CentOS and also here to Red Hat. It was already known at Red Hat and fixed there in coreutils-8.22-13.el7.

That one is not available yet for CentOS at this time (2015-08-20).

For completeness, note that the bug was also (incorrectly as the bug was not there) reported upstreams (at GNU's) where you'll find some more information about it.

Improve this answer
1

Your default collation(en_US.UTF-8) cause to this. You should set LC_COLLATE value to order the text as directed.

 LC_COLLATE='C' sort -k1 file1
Improve this answer
4
  • 1
    My question is why my default collation works this way? One of these can be true a is before A or a is after A not both. How can a collation result in both?
    – a5hk
    Aug 19, 2015 at 11:11
  • 1
    Not sort -k1, sort -k1 is the same as sort. It sorts on the section of the line that starts at the first field, so basically the whole line. sort -k1,1 to sort on the first field.
    – Stéphane Chazelas
    Aug 19, 2015 at 11:32
  • @Ashkan I think a collation could declare that A and a are equal. The sort command would then sort by later characters on the line, or failing that, leave them in the order they originally appeared in the file. E.g. ac, aa, Ab, should be sorted as aa, Ab, ac , not Ab, aa, ac or aa, ac, Ab.
    – rjmunro
    Apr 28, 2020 at 9:31
  • FYI neither LC_ALL=C nor LC_COLLATE=C helped solve the problem on my side: (1) # export LC_ALL=C # printf "%s\n" "1000|a" "103|a" "1034|a" | sort --field-separator='|' --key=1 outputs: 1000|a 1034|a 103|a (2) # export LC_COLLATE=C # printf "%s\n" "1000|a" "103|a" "1034|a" | sort --field-separator='|' --key=1 outputs: 1000|a 1034|a 103|a where the expected order is 1000|a 103|a 1034|a
    – bernard paulus
    Jun 20, 2021 at 13:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .