Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am having some issues with a script that is using join to join two files. Eaxmple input files contains lines like this:

Here are the input file and the output of the join command:

D:\work\BuildScripts\3C>cat D:\temp\aaa.txt
hzapplications\adn\adn4\adn4density\adn4_idd_module.cpp,83
hzapplications\adn\adn4\adn4density\adn4dencalmodule.cpp,73
hzapplications\adn\adn4\adn4density\adn4denimagemodulerm.cpp,111
hzapplications\adn\adn4\adn4density\adn4denimagemodulert.cpp,202
hzapplications\adn\adn4\adn4density\adn4densityanqmodules.cpp,445
hzapplications\adn\adn4\adn4density\adn4densityappl.cpp,378
hzapplications\adn\adn4\adn4density\adn4densityappl.h,50
hzapplications\adn\adn4\adn4density\adn4densityevrmodules.cpp,272
hzapplications\adn\adn4\adn4density\adn4densitykernel.cpp,490
hzapplications\adn\adn4\adn4density\adn4densitykernel.h,65
hzapplications\adn\adn4\adn4density\adn4densitysecimgmodule.cpp,209
hzapplications\adn\adn4\adn4density\adn4densitysecimgmodule.h,70
hzapplications\adn\adn4\adn4density\adn4densitysecmodule.cpp,218
hzapplications\adn\adn4\adn4density\adn4densitysecmodule.h,70
hzapplications\adn\adn4\adn4density\adn4dphimodules.cpp,610
hzapplications\adn\adn4\adn4density\adn4dphimodulesrt.cpp,115
hzapplications\adn\adn4\adn4density\adn4rhomodulesrt.cpp,102

D:\work\BuildScripts\3C>cat D:\temp\bbb.txt
hzapplications\activect\ptc\ictsx01\ictsx01_bootuptask.cpp,1
hzapplications\activeps\iola\acquisition\iola_acqmodule.cpp,4
hzapplications\activeps\iola\simulation\iola_simmodule.cpp,3
hzapplications\activeps\iolr\simulation\iolr_simmodule.cpp,1
hzapplications\activeps\iolr\task\iolr_poweron200vhitask.cpp,1
hzapplications\activeps\iolr\task\iolr_poweron200vlowtask.cpp,1
hzapplications\activeps\iolr\task\iolr_poweronnrlvtask.cpp,1
hzapplications\activeps\iolr\task\iolrtaskcommon.cpp,2
hzapplications\adn\adn4\adn4density\adn4densitykernel.cpp,1
hzapplications\adn\adn4\adn4equipment\adn4adseelem.cpp,1
hzapplications\adn\adn4\adn4equipment\adn4collar.cpp,1
hzapplications\adn\adn4\adn4equipment\adn4tool.cpp,2
hzapplications\adn\adn6c\adn6cequipment\adn6ccollar.cpp,1
hzapplications\adn\adn8\adn8equipment\adn8tool.cpp,1
hzapplications\adn\adn8\adn8neutron\adn8neutronkernel.cpp,1
hzapplications\adn\adn8d\adn8ddensity\adn8ddensitykernel.cpp,1
hzapplications\adn\adn8d\adn8dequipment\adn8dtool.cpp,1

D:\work\BuildScripts\3C>join --ignore-case -1 1 -2 1 -t"," -o "1.1,1.2,2.2" -e "0" -a 1 D:\temp\aaa.txt D:\temp\bbb.txt
hzapplications\adn\adn4\adn4density\adn4_idd_module.cpp,83,0
hzapplications\adn\adn4\adn4density\adn4dencalmodule.cpp,73,0
hzapplications\adn\adn4\adn4density\adn4denimagemodulerm.cpp,111,0
hzapplications\adn\adn4\adn4density\adn4denimagemodulert.cpp,202,0
hzapplications\adn\adn4\adn4density\adn4densityanqmodules.cpp,445,0
hzapplications\adn\adn4\adn4density\adn4densityappl.cpp,378,0
hzapplications\adn\adn4\adn4density\adn4densityappl.h,50,0
hzapplications\adn\adn4\adn4density\adn4densityevrmodules.cpp,272,0
hzapplications\adn\adn4\adn4density\adn4densitykernel.cpp,490,0
hzapplications\adn\adn4\adn4density\adn4densitykernel.h,65,0
hzapplications\adn\adn4\adn4density\adn4densitysecimgmodule.cpp,209,0
hzapplications\adn\adn4\adn4density\adn4densitysecimgmodule.h,70,0
hzapplications\adn\adn4\adn4density\adn4densitysecmodule.cpp,218,0
hzapplications\adn\adn4\adn4density\adn4densitysecmodule.h,70,0
hzapplications\adn\adn4\adn4density\adn4dphimodules.cpp,610,0
hzapplications\adn\adn4\adn4density\adn4dphimodulesrt.cpp,115,0
hzapplications\adn\adn4\adn4density\adn4rhomodulesrt.cpp,102,0

D:\work\BuildScripts\3C>

The expected output is that this particular line is joined like so: hzapplications\adn\adn4\adn4density\adn4densitykernel.cpp,490,1

Any suggests are most welcome. I am using the unxutils package on windows, this is the exact version:

D:\work\BuildScripts\3C>join --version
join (GNU textutils) 2.0
Written by Mike Haertel.

Copyright (C) 1999 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
share|improve this question
    
I've only skimmed this but... are your files sorted? join doesn't work if the input files aren't sorted on the join field. –  Alan Curry Aug 30 '12 at 6:43
    
Hi Alan, thanks for the suggestion. Yes, the input files should be sorted, but to make double sure I have updated the test script to include a sort before doing the join. –  mikelong Aug 30 '12 at 8:12
    
This is a very old version of join, can you try a newer version? –  Thor Aug 30 '12 at 12:56
3  
That's not really sorting on the join field. You need sort -t ',' -k 1,1 which could produce different results for some inputs. Recent versions of GNU coreutils have a join --check-order option that would help. It would be easier to see what's wrong if you just posted the actual aaa and bbb files for your failing test case. All the stuff leading up to their creation is irrelevant, boring, and not reproducible by us anyway. –  Alan Curry Aug 30 '12 at 20:52
    
Hi Alan, thanks for pointing me towards the correct sort command. I have edited the question to add the raw input files and the output of the join command. –  mikelong Aug 31 '12 at 2:43
show 2 more comments

1 Answer 1

up vote 2 down vote accepted

It turns out that --ignore-case is the problem. It has an effect even when there are no uppercase letters because it treats all lowercase letters as uppercase, causing them to jump to the other side of the characters that are between the uppercase and lowercase in ASCII order: [\]^_

In normal sorted order, iolrt comes after iolr_ but in --ignore-case order they are reversed.

The sort command needs the -f option to produce the correct order. (In addition to -t, and -k1,1)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.