275

When I open the file in Vim, I see strange ^M characters.

Unfortunately, the world's favorite search engine does not do well with special characters in queries, so I'm asking here:

  • What is this ^M character?

  • How could it have got there?

  • How do I get rid of it?

3
  • 4
    It's a control character digraph, for more info: help digraph-table. Jul 1, 2019 at 2:50
  • VS Code has an EOL option at the bottom that will auto convert an open file. Apr 6, 2020 at 20:06
  • @StackUnderflow when using VSCode, changing the line endings to LF when working on linux files helps. I use WSL ...A Windows file like $profile has no problem with Linux file ending LF. Maybe Win beats Linux here.
    – Timo
    Nov 14, 2020 at 8:37

13 Answers 13

191

The ^M is a carriage-return character. If you see this, you're probably looking at a file that originated in the DOS/Windows world, where an end-of-line is marked by a carriage return/newline pair, whereas in the Unix world, end-of-line is marked by a single newline.

Read this article for more detail, and also the Wikipedia entry for newline.

This article discusses how to set up vim to transparently edit files with different end-of-line markers.

If you have a file with ^M at the end of some lines and you want to get rid of them, use this in Vim:

:s/^M$//

(Press Ctrl+V Ctrl+M to insert that ^M.)

11
  • 21
    Try :%s/^M/\r/g instead to remove ^M and replace ^M with newline character \r. Without %, the command applies for current line only. And I came across some examples where ^M is not at end of line, such as The first line.^MThe second line.
    – George
    Apr 14, 2015 at 4:29
  • 3
    Or if you don't want loads of line breaks you could just do :%s/^M/ Nov 21, 2016 at 8:52
  • 2
    If it's just a carriage return by itself, that might be the classic (pre-Unix) Macintosh line break. Even some newer programs like Excel 2007 for Mac do that for some reason.
    – sudo
    Apr 18, 2017 at 17:22
  • 5
    Not sure if it's something in my vim config, but I have to type Ctrl+Q, then Ctrl+M to get that character to generate!
    – Brad Parks
    Jan 14, 2020 at 19:59
  • 1
    @BradParks you are probably using (sourcing) mswin.vim. It's the default on Windows. Nov 22, 2021 at 17:50
86

A simpler way to do this is to use the following command:

dos2unix filename

This command works with path patterns as well, Eg

dos2unix path/name*

If it doesn't work, try using different mode:

dos2unix -c mac filename
  • -c Set conversion mode. Where CONVMODE is one of: ascii, 7bit, iso, mac with ascii being the default.
2
  • 2
    +1 for mentioning about the -c flag default value and other options. At first try, the default didn't have the expected results. With dos2unix -c max it worked. dos2unix: converting file launch_me.sh to Unix format...
    – Johnny C
    Dec 19, 2019 at 16:43
  • Amazing little utility. For ubuntu just apt-get install dos2unix. If you need to use it recursively you can refer to this solution here: stackoverflow.com/questions/11929461/… I foolishly ran it on every directory manually and only thought to look up after the fact. Jul 22, 2020 at 5:36
80

This worked for me

:e ++ff=dos 

The :e ++ff=dos command tells Vim to read the file again, forcing dos file format. Vim will remove CRLF and LF-only line endings, leaving only the text of each line in the buffer.

then

:set ff=unix 

and finally

:wq 
6
  • 12
    thanks! this worked for me but the accepted solution didnt
    – 3pitt
    May 30, 2018 at 18:03
  • 1
    Does ff refers to file format? What about ++ff?
    – KcFnMi
    Aug 23, 2018 at 18:10
  • 1
    The :e ++ff=dos command tells Vim to read the file again, forcing dos file format. Vim will remove CRLF and LF-only line endings, leaving only the text of each line in the buffer.
    – Stryker
    Aug 29, 2018 at 20:05
  • 1
    To learn about ff, execute ":help ff"
    – sqqqrly
    Jan 22, 2020 at 15:03
  • simple and effective. I am not sure why cat file1 >> file2 is introducing dos line feeds when file 1 didn't have it in the first place Apr 23, 2020 at 12:09
48

Most UNIX operating systems have a utility called dos2unix that will convert the CRLF to LF. The other answers cover the "what are they" question.

12

Sed in-place solution without needing to type the special character (you can copy this and it works):

sed -i -e "s/\r//g" filename

Explanation:

-i: in-place
-e: regular expression
\r: escaped carriage return
/g: replace globally
7
  • The questioner was already editing the file with VIM, note. And there has been an in-place sed answer here since 2012.
    – JdeBP
    Mar 13, 2020 at 10:50
  • Yeah, but a good, working solution was missing. The existing sed solution doesn't work out-of-the-box. Mar 14, 2020 at 11:11
  • Dale Hagglund's in-place sed most definitely does work out of the box.
    – JdeBP
    Mar 16, 2020 at 10:09
  • 1
    Well, not for me. I'm on Ubuntu LTS 18.04. Mar 16, 2020 at 11:42
  • 2
    This helped me out editing a file on Alpine Linux from a Windows desktop that would count Ctrl M as a line return, making the above answers ineffective.
    – zacran
    Aug 13, 2020 at 23:26
11

Another way to get rid of carriage returns is with the tr command.

I have a small script that look like this

#!/bin/sh
tmpfile=$(mktemp)
tr -d '\r' <"$1" >"$tmpfile"
mv "$tmpfile" "$1"
9

You can clean this up with sed:

sed -e 's/^M$//' < infile > outfile

The trick is how to enter the carriage-return properly. Generally, you need to type C-v C-m to enter a literal carriage return. You can also have sed work in place with

sed -i.bak -e 's/^M$//' infile
7
  • doesn't work (macOS)
    – Dorian
    Feb 21, 2021 at 20:46
  • @Dorian Which part? The basic structure of the sed script is almost certain to work, but entering the literal CR (^M) depends on which terminal program you're using, and I don't know anything about Mac's in that regard. Feb 22, 2021 at 4:59
  • @Dorian To figure out what's not working, start by replacing the ^M with, eg, a capital A, and see if the sed command strips a final A from its input lines. If no, you're getting your basic sed syntax wrong somehow; if yes, you're having trouble correctly typing the single CR character into the shell. Feb 22, 2021 at 5:02
  • sorry I don't have the file to test it anymore, not sure, maybe it's because I use fish, maybe sed is different on macOS, etc.
    – Dorian
    Feb 22, 2021 at 9:59
  • @Dorian No idea about fish, but I'm pretty sure you could create your own test file of one or two lines. Feb 22, 2021 at 10:39
5

What is this ^M?
The ^M is a carriage-return character. If you see this, you're probably looking at a file that originated in the DOS/Windows world, where an end-of-line is marked by a carriage return/newline pair, whereas in the Unix world, end-of-line is marked by a single newline.

How could it have got there?
When there is change in file format.

How do I get rid of it?
open your file with

vim -b FILE_PATH

save it with following command

:%s/^M//g
2
  • You've got a typo in open your filr with. Jan 27, 2016 at 20:48
  • 5
    This answer does not add anything to the other answers. The first paragraph is an almost verbatim copy from the accepted answer. The given code will not save anything, but just remove all carriage return characters from all lines. And I am not sure how opening the file in binary mode will help here.
    – Dubu
    Jan 28, 2016 at 7:58
3

You can use Vim in Ex mode:

ex -bsc '%s/\r//|x' file
  1. -b binary mode

  2. % select all lines

  3. s substitute

  4. \r carriage return

  5. x save and close

2

In my case,

Nothing above worked, I had a CSV file copied to Linux machine from my mac and I used all the above commands but nothing helped but the below one

tr "\015" "\n" < inputfile > outputfile

I had a file in which ^M characters were sandwitched between lines something like below

Audi,A4,35 TFSi Premium,,CAAUA4TP^MB01BNKT6TG,TRO_WBFB_500,Trico,CARS,Audi,A4,35 TFSi Premium,,CAAUA4TP^MB01BNKTG0A,TRO_WB_T500,Trico,
1
  • mac2unix -n inputfile outputfile, or, equivalently, dos2unix -c mac -n inputfile outputfile will handle that situation. Dec 11, 2019 at 1:48
0

Add the following line to your ~/.vimrc

command! Tounix :call Preserve('1,$s/^M//')

Then when you have a file with the Windows line endings, run the command ":Tounix".

0

If your file uses mixed line breaks, i. e. \r\n and \r, you can use this sed script (this is an all-in-one solution and of course, you can also use it if your file merely has \r\n or \r line breaks):

sed -i[SUFFIX] ':read; N; $!b read; s/\r\n/\n/g; s/\r/\n/g' file

-i tells sed to replace your file with the result of the script, and if you supply a SUFFIX, a backup will be created with that suffix.

You also may omit -i and redirect the output to an arbitrary file.

How the script works:

Special commands:

  • N: Adds a newline to the pattern space, then appends the next line of input (with any trailing \n removed) to the pattern space.
  • $!: Don't execute the following command on the last line.
  • b: Branches unconditionally to the specified label.

So when the cycle starts (we have just one cycle here), sed reads the first line of input, removes any trailing \n and places it in the pattern space, then it processes the script:

N adds a newline to the pattern space, then appends the next line of input (with any trailing \n removed) to the pattern space. This is repeated until the last line is reached, then the substitutions operate on the pattern space, which contains the whole file as a single line, which is the reason why we need the g flag for the substitutions.

The first substitution replaces each \r\n with \n. After this, the remaining ("lonesome") \rs are replaced with \n.

Why do we need the N loop?

Since sed 's/<regexp>/<replacement>/[flags]' file reads a line from the file, performs the substitution on it, prints the result, reads the next line and so on, \n cannot be used within the <regexp> part.

But if we have the whole file as a single line in the pattern space, we will be able to use \n within the <regexp> part, and this is necessary to distinguish between the sequence \r\n and "lonesome" \rs.

Example:

$ sudo file /var/log/apt/term.log
/var/log/apt/term.log: UTF-8 Unicode text, with CRLF, CR, LF line terminators, with escape sequences, with overstriking
$ sudo sed -i.bak ':read; N; $!b read; s/\r\n/\n/g; s/\r/\n/g' /var/log/apt/term.log
$ sudo file /var/log/apt/term.log
/var/log/apt/term.log: UTF-8 Unicode text, with escape sequences, with overstriking
-2

In the past, I have seen even configuration files are not parsed properly and complain about whitespace, but if you vi and do a set list it won't show the whitespace, grep filename [[space]] will show you ^M

that's when dos2unix file helps

1
  • This answer does not bring nothing new, does2unix is already mentioned at least in two answers much more older. Apr 12, 2019 at 14:06

Not the answer you're looking for? Browse other questions tagged or ask your own question.