238

When I open the file in Vim, I see strange ^M characters.

Unfortunately, the world's favorite search engine does not do well with special characters in queries, so I'm asking here:

  • What is this ^M character?

  • How could it have got there?

  • How do I get rid of it?

3
  • 3
    It's a control character digraph, for more info: help digraph-table. – Eric Leschinski Jul 1 '19 at 2:50
  • VS Code has an EOL option at the bottom that will auto convert an open file. – Stack Underflow Apr 6 '20 at 20:06
  • @StackUnderflow when using VSCode, changing the line endings to LF when working on linux files helps. I use WSL ...A Windows file like $profile has no problem with Linux file ending LF. Maybe Win beats Linux here. – Timo Nov 14 '20 at 8:37

12 Answers 12

164

The ^M is a carriage-return character. If you see this, you're probably looking at a file that originated in the DOS/Windows world, where an end-of-line is marked by a carriage return/newline pair, whereas in the Unix world, end-of-line is marked by a single newline.

Read this article for more detail, and also the Wikipedia entry for newline.

This article discusses how to set up vim to transparently edit files with different end-of-line markers.

If you have a file with ^M at the end of some lines and you want to get rid of them, use this in Vim:

:s/^M$//

(Press Ctrl+V Ctrl+M to insert that ^M.)

8
  • 18
    Try :%s/^M/\r/g instead to remove ^M and replace ^M with newline character \r. Without %, the command applies for current line only. And I came across some examples where ^M is not at end of line, such as The first line.^MThe second line. – George Apr 14 '15 at 4:29
  • 3
    Or if you don't want loads of line breaks you could just do :%s/^M/ – carefulnow1 Nov 21 '16 at 8:52
  • 2
    If it's just a carriage return by itself, that might be the classic (pre-Unix) Macintosh line break. Even some newer programs like Excel 2007 for Mac do that for some reason. – sudo Apr 18 '17 at 17:22
  • @larsks But altap.cz/salamander/help/salamand/appendix_txtfiles says that ^M is used in UNIX – vivex Feb 16 '18 at 7:23
  • 2
    Not sure if it's something in my vim config, but I have to type Ctrl+Q, then Ctrl+M to get that character to generate! – Brad Parks Jan 14 '20 at 19:59
74

A simpler way to do this is to use the following command:

dos2unix filename

This command works with path patterns as well, Eg

dos2unix path/name*

If it doesn't work, try using different mode:

dos2unix -c mac filename
  • -c Set conversion mode. Where CONVMODE is one of: ascii, 7bit, iso, mac with ascii being the default.
2
  • 2
    +1 for mentioning about the -c flag default value and other options. At first try, the default didn't have the expected results. With dos2unix -c max it worked. dos2unix: converting file launch_me.sh to Unix format... – Johnny C Dec 19 '19 at 16:43
  • Amazing little utility. For ubuntu just apt-get install dos2unix. If you need to use it recursively you can refer to this solution here: stackoverflow.com/questions/11929461/… I foolishly ran it on every directory manually and only thought to look up after the fact. – anon58192932 Jul 22 '20 at 5:36
62

This worked for me

:e ++ff=dos 

The :e ++ff=dos command tells Vim to read the file again, forcing dos file format. Vim will remove CRLF and LF-only line endings, leaving only the text of each line in the buffer.

then

:set ff=unix 

and finally

:wq 
5
  • 9
    thanks! this worked for me but the accepted solution didnt – 3pitt May 30 '18 at 18:03
  • 1
    Does ff refers to file format? What about ++ff? – KcFnMi Aug 23 '18 at 18:10
  • 1
    The :e ++ff=dos command tells Vim to read the file again, forcing dos file format. Vim will remove CRLF and LF-only line endings, leaving only the text of each line in the buffer. – Stryker Aug 29 '18 at 20:05
  • 1
    To learn about ff, execute ":help ff" – sqqqrly Jan 22 '20 at 15:03
  • simple and effective. I am not sure why cat file1 >> file2 is introducing dos line feeds when file 1 didn't have it in the first place – abacusreader Apr 23 '20 at 12:09
46

Most UNIX operating systems have a utility called dos2unix that will convert the CRLF to LF. The other answers cover the "what are they" question.

11

Another way to get rid of carriage returns is with the tr command.

I have a small script that look like this

#!/bin/sh
tmpfile=$(mktemp)
tr -d '\r' <"$1" >"$tmpfile"
mv "$tmpfile" "$1"
9

You can clean this up with sed:

sed -e 's/^M$//' < infile > outfile

The trick is how to enter the carriage-return properly. Generally, you need to type C-v C-m to enter a literal carriage return. You can also have sed work in place with

sed -i.bak -e 's/^M$//' infile
7
  • doesn't work (macOS) – Dorian Feb 21 at 20:46
  • @Dorian Which part? The basic structure of the sed script is almost certain to work, but entering the literal CR (^M) depends on which terminal program you're using, and I don't know anything about Mac's in that regard. – Dale Hagglund Feb 22 at 4:59
  • @Dorian To figure out what's not working, start by replacing the ^M with, eg, a capital A, and see if the sed command strips a final A from its input lines. If no, you're getting your basic sed syntax wrong somehow; if yes, you're having trouble correctly typing the single CR character into the shell. – Dale Hagglund Feb 22 at 5:02
  • sorry I don't have the file to test it anymore, not sure, maybe it's because I use fish, maybe sed is different on macOS, etc. – Dorian Feb 22 at 9:59
  • @Dorian No idea about fish, but I'm pretty sure you could create your own test file of one or two lines. – Dale Hagglund Feb 22 at 10:39
9

Sed in-place solution without needing to type the special character (you can copy this and it works):

sed -i -e "s/\r//g" filename

Explanation:

-i: in-place
-e: regular expression
\r: escaped carriage return
/g: replace globally
7
  • The questioner was already editing the file with VIM, note. And there has been an in-place sed answer here since 2012. – JdeBP Mar 13 '20 at 10:50
  • Yeah, but a good, working solution was missing. The existing sed solution doesn't work out-of-the-box. – DreamFlasher Mar 14 '20 at 11:11
  • Dale Hagglund's in-place sed most definitely does work out of the box. – JdeBP Mar 16 '20 at 10:09
  • 1
    Well, not for me. I'm on Ubuntu LTS 18.04. – DreamFlasher Mar 16 '20 at 11:42
  • 2
    This helped me out editing a file on Alpine Linux from a Windows desktop that would count Ctrl M as a line return, making the above answers ineffective. – zacran Aug 13 '20 at 23:26
5

What is this ^M?
The ^M is a carriage-return character. If you see this, you're probably looking at a file that originated in the DOS/Windows world, where an end-of-line is marked by a carriage return/newline pair, whereas in the Unix world, end-of-line is marked by a single newline.

How could it have got there?
When there is change in file format.

How do I get rid of it?
open your file with

vim -b FILE_PATH

save it with following command

:%s/^M//g
2
  • You've got a typo in open your filr with. – Mateusz Piotrowski Jan 27 '16 at 20:48
  • 5
    This answer does not add anything to the other answers. The first paragraph is an almost verbatim copy from the accepted answer. The given code will not save anything, but just remove all carriage return characters from all lines. And I am not sure how opening the file in binary mode will help here. – Dubu Jan 28 '16 at 7:58
3

You can use Vim in Ex mode:

ex -bsc '%s/\r//|x' file
  1. -b binary mode

  2. % select all lines

  3. s substitute

  4. \r carriage return

  5. x save and close

2

In my case,

Nothing above worked, I had a CSV file copied to Linux machine from my mac and I used all the above commands but nothing helped but the below one

tr "\015" "\n" < inputfile > outputfile

I had a file in which ^M characters were sandwitched between lines something like below

Audi,A4,35 TFSi Premium,,CAAUA4TP^MB01BNKT6TG,TRO_WBFB_500,Trico,CARS,Audi,A4,35 TFSi Premium,,CAAUA4TP^MB01BNKTG0A,TRO_WB_T500,Trico,
1
  • mac2unix -n inputfile outputfile, or, equivalently, dos2unix -c mac -n inputfile outputfile will handle that situation. – Robin A. Meade Dec 11 '19 at 1:48
0

Add the following line to your ~/.vimrc

command! Tounix :call Preserve('1,$s/^M//')

Then when you have a file with the Windows line endings, run the command ":Tounix".

-2

In the past, I have seen even configuration files are not parsed properly and complain about whitespace, but if you vi and do a set list it won't show the whitespace, grep filename [[space]] will show you ^M

that's when dos2unix file helps

1
  • This answer does not bring nothing new, does2unix is already mentioned at least in two answers much more older. – Rui F Ribeiro Apr 12 '19 at 14:06

Not the answer you're looking for? Browse other questions tagged or ask your own question.