Sed processed file displays differently in vi vs cat

Question

I can't tell if sed is mucking my file up. In vi or less it displays properly, but cat and more insert other characters. why are they showing up differently

I am on a redhat linux system with a standard xterm.

the raw tab file before:

scaffold1000    693:14709284:741:333    129011535   1
scaffold1000    693:14709284:27:726 129011535   1
scaffold1000    693:14709284:44:1157    129011535   1
scaffold1000    693:14709284:771:459    129011535   1
scaffold1000    693:14709284:610:615    129011535   1
scaffold1000    693:14709284:1152:1159  129011535   1

applying sed:

sed -i 's/scaffold/scaffold\_/' [myfile]

I've also tried this without the backslash to the same result.

Using cat I see this :

scaffold11000   693:14709284:741:333    129011535   1
scaffold11000   693:14709284:27:726 129011535   1
scaffold11000   693:14709284:44:1157    129011535   1
scaffold11000   693:14709284:771:459    129011535   1
scaffold11000   693:14709284:610:615    129011535   1

Where did that '1' come from? :(

editing in vi or using less I see:

scaffold_1000   693:14709284:741:333    129011535       1^M     1^M     1       
scaffold_1000   693:14709284:27:726     129011535       1^M     1^M     1       
scaffold_1000   693:14709284:44:1157    129011535       1^M     1^M     1       
scaffold_1000   693:14709284:771:459    129011535       1^M     1^M     1       
scaffold_1000   693:14709284:610:615    129011535       1^M     1^M     1       
scaffold_1000   693:14709284:1152:1159  129011535       1^M     1^M     1    

Do the ^M chars have something to do with this? Its like I can't trust my own eyes here...

Answer 1

Each line of the file contains the string

^M     1

twice. That is,

<carriage-return><tab>1
^M^I1
\r\t1

(Those are three different representations of the same control characters)

When this is sent directly to your terminal, as cat will do, the terminal interprets this as an instruction to move back to the beginning of the line, move over 8 characters, and display a 1.

When you cat the original file, this additional 1 shows up on top of an existing 1, so you didn't notice it.

After you changed the prefix from scaffold to scaffold_, the additional 1 now shows up over the _, so now you noticed it.

When you open the file in vi or emacs, the editor doesn't interpret the <carriage-return> sequence the same way, but instead displays it.

I assume you don't want the <carriage-return><tab>1 strings in your file. You can remove them with sed:

sed -i 's/\r\t1//g' myfile

---

Regarding your "I can't trust my own eyes" comment -- since control sequences change how things get displayed, yeah, they can be confusing. One way to view things more clearly is with cat's -v and -T options:

-v, --show-nonprinting   use ^ and M- notation, except for LFD and TAB
-T, --show-tabs          display TAB characters as ^I
-t                       equivalent to -vT

For example:

$ cat myfile
scaffold1test
$ cat -t myfile
scaffold_hello^M^I1

Answer 2

If you don't have dos2unix, you can do it with sed :

sed -i 's/\r//g' <THE FILE NAME>

It will remove all carriage returns.

Answer 3

^M is a carriage return.

you can use dos2unix <file> to strip the DOS keys out of your file