How to fix unix `join` command inserting ^M between join columns? -Unix

Question

I am a novice programmer. I am using unix's join command to self-join couple long files together.

join -j30 test test2
col1 col2 ... col30    col1 col2 ... col30
A    B        ZZZ   ^M A    B        ZZZ

I am getting this ^M character in my output.

Why is it there? and How would I remove it?

EDIT: Below is a screenshot of my part of my output

Answer 1

The ^M means you are bringing over/editing file in Windows.

Use the dos2unix command over the files to convert them to Unix text mode.

DOS uses carriage return and line feed "\r\n" as a line ending, while Unix uses just line feed "\n". The ^M are a visual representation of the "extra" \r characters.

To install the dos2unix command, do (on Debian-based distros):

sudo apt-get install dos2unix

or in a Mac (MacPorts):

sudo port install dos2unix

Alternatively, you can also do it with sed as in:

sed 's/\r$//' dosfile.txt > unixfile.txt

Answer 2

join(1) (at least the version I'm using) does not add ^M or carriage returns to joined files; join will however happily copy ^M along. A hex viewer plus some test cases can show what is going on:

$ echo "1 a" > test1
$ echo "1 b" > test2
$ join test1 test2 | od -bc
0000000   061 040 141 040 142 012                                        
           1       a       b  \n                                        
0000006
$

And a test case where there are ^M in the input files:

$ perl -E "say qq(1 a\r)" > test1
$ perl -E "say qq(1 b\r)" > test2
$ join test1 test2 | od -bc
0000000   061 040 141 015 040 142 015 012                                
           1       a  \r       b  \r  \n                                
0000010
$ 

There are various ways to correct the input files, besides dos2unix one could also use tr (though this would be a terrible idea if the file uses the ancient MacOS tradition of \r as a line delimiter):

$ od -bc test1
0000000   061 040 141 015 012                                            
           1       a  \r  \n                                            
0000005
$ tr -d '\r' < test1 > foo
$ mv foo test1
$ od -bc test1            
0000000   061 040 141 012                                                
           1       a  \n                                                
0000004
$