Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I can't tell if sed is mucking my file up. In vi or less it displays properly, but cat and more insert other characters. why are they showing up differently

I am on a redhat linux system with a standard xterm.

the raw tab file before:

scaffold1000    693:14709284:741:333    129011535   1
scaffold1000    693:14709284:27:726 129011535   1
scaffold1000    693:14709284:44:1157    129011535   1
scaffold1000    693:14709284:771:459    129011535   1
scaffold1000    693:14709284:610:615    129011535   1
scaffold1000    693:14709284:1152:1159  129011535   1

applying sed:

sed -i 's/scaffold/scaffold\_/' [myfile]

I've also tried this without the backslash to the same result.

Using cat I see this :

scaffold11000   693:14709284:741:333    129011535   1
scaffold11000   693:14709284:27:726 129011535   1
scaffold11000   693:14709284:44:1157    129011535   1
scaffold11000   693:14709284:771:459    129011535   1
scaffold11000   693:14709284:610:615    129011535   1

Where did that '1' come from? :(

editing in vi or using less I see:

scaffold_1000   693:14709284:741:333    129011535       1^M     1^M     1       
scaffold_1000   693:14709284:27:726     129011535       1^M     1^M     1       
scaffold_1000   693:14709284:44:1157    129011535       1^M     1^M     1       
scaffold_1000   693:14709284:771:459    129011535       1^M     1^M     1       
scaffold_1000   693:14709284:610:615    129011535       1^M     1^M     1       
scaffold_1000   693:14709284:1152:1159  129011535       1^M     1^M     1    

Do the ^M chars have something to do with this? Its like I can't trust my own eyes here...

share|improve this question
    
I can't see how that have happend. Can you start with a fresh file and check that you can repeat that behaviour? –  January Oct 9 '12 at 18:56
    
I think @JimParis nailed this - take a look below. this is definitely reproducible. its just odd how the formatting conspires to make exactly the same change in all the lines I looked at. –  shigeta Oct 9 '12 at 22:27

3 Answers 3

up vote 7 down vote accepted

Each line of the file contains the string

^M     1

twice. That is,

<carriage-return><tab>1
^M^I1
\r\t1

(Those are three different representations of the same control characters)

When this is sent directly to your terminal, as cat will do, the terminal interprets this as an instruction to move back to the beginning of the line, move over 8 characters, and display a 1.

When you cat the original file, this additional 1 shows up on top of an existing 1, so you didn't notice it.

After you changed the prefix from scaffold to scaffold_, the additional 1 now shows up over the _, so now you noticed it.

When you open the file in vi or emacs, the editor doesn't interpret the <carriage-return> sequence the same way, but instead displays it.

I assume you don't want the <carriage-return><tab>1 strings in your file. You can remove them with sed:

sed -i 's/\r\t1//g' myfile

Regarding your "I can't trust my own eyes" comment -- since control sequences change how things get displayed, yeah, they can be confusing. One way to view things more clearly is with cat's -v and -T options:

-v, --show-nonprinting   use ^ and M- notation, except for LFD and TAB
-T, --show-tabs          display TAB characters as ^I
-t                       equivalent to -vT

For example:

$ cat myfile
scaffold1test
$ cat -t myfile
scaffold_hello^M^I1
share|improve this answer
    
this totally makes sense. thanks –  shigeta Oct 9 '12 at 22:25

^M is a carriage return.

you can use dos2unix <file> to strip the DOS keys out of your file

share|improve this answer
    
dos2unix will convert line endings from CRLF to LF. There are no CRLF sequences in this file; just bare CRs (not at the end of lines). –  Jim Paris Oct 9 '12 at 17:57
    
how did you know that just by looking at the excerpt from his output? –  h3rrmiller Oct 9 '12 at 18:02
1  
From what he copied out of vi and less. They showed the bare CRs directly in the middle of the lines. Also, it fully explains the behavior he's seeing. –  Jim Paris Oct 9 '12 at 18:04

If you don't have dos2unix, you can do it with sed :

sed -i 's/\r//g' <THE FILE NAME>

It will remove all carriage returns.

share|improve this answer
    
This command isn't equivalent to dos2unix. dos2unix strips the CR from a CRLF; this command will strip CR everywhere. But the asker's file has more than just stray CRs, so this will leave it with the extraneous TAB and 1 characters too. –  Jim Paris Oct 9 '12 at 17:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.