Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've got two files _jeter3.txt and _jeter1.txt

I've checked they are both sorted on the 20th column using sort -c

sort -t '     ' -c -k20,20 _jeter3.txt
sort -t '     ' -c -k20,20 _jeter1.txt
#no errors

but there is an error when I want to join both files it says that the second file is not sorted:

join -t '   ' -1 20 -2 20 _jeter1.txt _jeter3.txt > /dev/null
join: File 2 is not in sorted order

I don't understand why.

cat /etc/*-release #FYI
openSUSE 11.0 (i586)
VERSION = 11.0

UPDATE: using 'sort -f' and join -i (both case insensitive) fixes the problem. But it doesn't explain my initial problem.

UPDATE: versions of sort & join:

> join --version
join (GNU coreutils) 6.11
Copyright (C) 2008 Free Software Foundation, Inc.
(...)

> sort --version
sort (GNU coreutils) 6.11
Copyright (C) 2008 Free Software Foundation, Inc.
(...)
share|improve this question
    
Can you give us the output of "join --version" and "sort --version" just for completeness sake? I can't get some older versions of gnu join to give me that error message under any circumstance. –  Bruce Ediger May 10 '11 at 18:44
3  
Please post some sample data that exhibits the problem, and the output of locale. –  Gilles May 10 '11 at 22:37

5 Answers 5

up vote 12 down vote accepted

I got the same error with Ubuntu 11.04, with sort and join both in version (GNU coreutils) 8.5.

They are clearly incompatible. In fact the sort command seems bugged: there is no difference with or without the -f (--ignore-case) option. When sorting, aaB is always before aBa. Non alphanumeric characters seems also always ignored (abc is before ab-x)

Join seems to expect the opposite... But I have a solution

In fact, this is linked to the collation sequence : unsing LANG=en_EN sort -k 1,1 <myfile> ... then LANG=en_EN join ... worked.

Internationalisation is the root of evil... (nobody documents it clearly)

share|improve this answer
    
So, if both use LANG=en_EN, then it will definitely work? Will it work for any locale, as long as both use the same locale? Can we say that the difference between sort and join is that they use a different locale by default? –  Aaron McDaid Aug 25 at 15:19

sort by default uses the entire line as the key

join uses only the specified field as the key.

You must correct this incompatibility by restricting sort to use only the key you want to join on.

The Join man page states:

Important: FILE1 and FILE2 must be sorted on the join fields. E.g., use 'sort -k 1b,1' if >'join' has no options. Note, comparisons honor the rules specified by 'LC_COLLATE'. If the >input is not sorted and some lines cannot be joined, a warning message will be given.

share|improve this answer

If you are sure you properly sorted your input files and their lines can be paired, you can avoid the above error by running join --nocheck-order file1.txt file2.txt

share|improve this answer

Were you sorting with numbers? I found that zero-padding the column that I was joining on solved this issue for me.

cat file.txt \
     | awk -F"   " '{ $20=sprintf("%06s", $20); print $0}' \
     | sort > readytojoin.txt
share|improve this answer

For join the argument after -t is a character. For sort you can supply a longer sort separator. I think that you may be joining the files on a different field that you want to, and ignoring the case solves the problem by coincidence.

And I agreee with Gilles, that sample data would be helpful.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.