I'd like to remove all carriage returns followed by line feeds (CRLF), such as \r\n
in a file. How can I do that? I can't use dos2unix
because that replaces CRLF with LF. And I can't use tr
because that will also replace any \n
that aren't preceded by \r
. How can I do this?
sed ":a;/\r$/{N;s/\r\n//;b a}"
This will match all lines that have '\r'
at the end (followed by '\n'
). On these lines it will first append the next line of input (while putting the '\n
separator in between), then replace the resulting "\r\n"
with an empty string, and then goes back to the beginning to see, whether the new contents of pattern space doesn't by chance happen to match again.
Following the comment: if you wanted to strip any additional '\r'
from the file as well, just add it after stripping the CRLF combos:
sed ":a;/\r$/{$!N;s/\r\n//;t a};s/\r//g"
-
Cool! Btw is there a way to modify that command to strip all occurrences of
\r
in addition to\r\n
from the file? (Rather than having to run a second command to get rid of\r
using something liketr
.) – user779159 Jan 17 '15 at 15:13 -
Thanks peterph, it works great. (And mikeserv for an edit.) Putting 2 sed commands in the same command separated by a semicolon is more efficient than running them as 2 separate commands? Meaning it only has to scan the file through once and just runs both commands on each line? – user779159 Jan 17 '15 at 16:38
-
-
@mikeserv thanks for the edit. However the branch command needs modifying as well - the unconditional one I had there was causing an endless loop on the last line. – peterph Jan 17 '15 at 16:53
-
Maybe like
sed -e :n -e '/^M$/{$s///p;N' -e '};s/.\n//;tn'
The problem is though thatsed
isnt designed for unlimited line length. If youre relying on gnu extensions-z
is probably easiest:sed -z 's/\r\n//g'
. But then youre working with long pattern spaces. At lest that way though you can clear it without printing a newline - so once per edit. Andsed
probably doesnt save much here - im willing to bet an additionaltr
would actually save processing time. – mikeserv Jan 17 '15 at 17:13
I tend to reach for perl
one-liners when doing anything that involves manipulating line endings:
perl -pe 'BEGIN {undef $/} s/\r\n//g' *.txt
The key to making this work is the undef $/
, which makes Perl read each file as one string, which you can then do a search-and-replace on. To strip bare \r
as well, just tweak the regex:
perl -pe 'BEGIN {undef $/} s/\r\n?//g' *.txt
-
-
@AvinashRaj That is the same as
dos2unix
, which is specifically not what the OP wanted. – zwol Jan 18 '15 at 16:36
sed -i 's/\r\n//g' file
which didn't work – user779159 Jan 17 '15 at 14:49