How to test whether a file uses CRLF or LF without modifying it?

Question

I need to periodically run a command that ensures that some text files are kept in Linux mode. Unfortunately dos2unix always modifies the file, which would mess file's and folder's timestamps and cause unnecessary writes.

The script I write is in Bash, so I'd prefer answers based on Bash.

Answer 1

Note: This should not be used in automation scripts, just for checking quickly. For automation scripts, I'd suggest you look at other answers

Use cat -A

$ cat file
hello
hello

Now if this file was made in *NIX systems, it would display

$ cat -A file
hello$
hello$

But if this file was made in Windows, it would display

$ cat -A file
hello^M$
hello

^M represents CR and $ represents LF.

I used Windows Notepad in Windows and Vim in Linux. Notice that Windows Notepad did not save the last line with CRLF.

This does not change the file contents either.

Answer 2

You can use dos2unix as a filter and compare its output to the original file:

dos2unix < myfile.txt | cmp - myfile.txt

Answer 3

If the goal is just to avoid affecting the timestamp, dos2unix has a -k or --keepdate option which will keep the timestamp the same. It will still have to do a write to make the temporary file and rename it, but your timestamps will not be affected.

If any modification of the file is unacceptable, you can use the following solution from this answer.

find . -not -type d -exec file "{}" ";" | grep CRLF

The file command will output CRLF if it detects such line terminations.

Answer 4

Since version 7.1, dos2unix has an -i, --info option to get information about line breaks. Since version 7.3.5 (July 2017), the -i option supports a 0 flag to use a null separator. You can use dos2unix itself to test which files need conversion.

For example, assuming GNU xargs:

dos2unix -ic0 -- *.txt | xargs -r0 dos2unix --

would only convert the files that need converting (reported as such by dos2unix -ic).

Answer 5

You could try to grep for CRLF code, octal:

grep -U $'\015' myfile.txt

or hex:

grep -U $'\x0D' myfile.txt

Answer 6

First method (grep):

Count the lines that contain a carriage return:

[[ $(grep -c $'\r' myfile.txt) -gt 0 ]] && echo dos

Count the lines that end with a carriage return:

[[ $(grep -c $'\r$' myfile.txt) -gt 0 ]] && echo dos

These will typically be equivalent; a carriage return in the interior of a line (i.e., not at the end) is rare.

More efficient:

grep -q $'\r' myfile.txt && echo dos

This is more efficient

1. because it doesn't need to convert the count to an ASCII string, and then convert that string back to an integer, and compare it to zero, and
2. because grep -c needs to read the entire file, to count all the occurrences of the pattern, while grep -q can exit upon seeing the first occurrence of the pattern.

Notes:

• Throughout the above, you may need to add the -U option (i.e., use -cU or -qU), because GNU grep guesses whether the file is a text file.  If it thinks the file is text, it ignores carriage returns at the ends of lines, in an attempt to make $ in regular expressions work "correctly" — even if the regular expression is \r$!  Specifying -U (or --binary) overrules this guesswork, causing grep to treat the file(s) as binary and pass the data to the matching mechanism verbatim, with CR-endings intact.
• Do not do grep … $'\r\n' myfile.txt, because grep treats \n as a pattern delimiter.  Just as grep -E 'foo|' looks for lines containing foo or a null string, grep $'\r\n' looks for lines containing \r or a null string, and every line matches a null string.

Second method (file):

[[ $(file myfile.txt) =~ CRLF ]] && echo dos

because file reports something like:

myfile.txt: UTF-8 Unicode text, with CRLF line terminators

Safer variant:

[[ $(file -b - < myfile.txt) =~ CRLF ]] && echo dos

where

• file -b outputs only the file type, and not the file name.  Without this, a file whose name included the characters CRLF would trigger a false positive.
• file - < filename works even if filename begins with -.  ^{See Bash script: check if a file is a text file.}

Beware that checking the output from file might not work in a non-English locale.

Answer 7

Use file:

$ file README.md
README.md: ASCII text, with CRLF line terminators

$ dos2unix README.md
dos2unix: converting file README.md to Unix format...

$ file README.md
README.md: ASCII text

Answer 8

a bash function for you:

# return 0 (true) if first line ends in CR
isDosFile() {
    [[ $(head -1 "$1") == *$'\r' ]]  
}

Then you can do stuff like

streamFile () {
    if isDosFile /tmp/foo.txt; then
        sed 's/\r$//' "$1"
    else
        cat "$1"
    fi
}

streamFile /tmp/foo.txt | process_lines_without_CR

Answer 9

If a file has DOS/Windows-style CR-LF line endings, then if you look at it using a Unix-based tool you'll see CR ('\r') characters at the end of each line.

This command:

grep -l '^M$' filename

will print filename if the file contains one or more lines with Windows-style line endings, and will print nothing if it doesn't. Except that the ^M has to be a literal carriage return character, typically entered in the terminal by typing Ctrl+V followed by Enter (or Ctrl+V and then Ctrl+M). The bash shell lets you write a literal carriage return as $'\r' (documented here), so you can write:

grep -l $'\r$' filename

Other shells may provide a similar feature.

You can use another tool instead:

awk '/\r$/ { exit(1) }' filename

This will exit with a status of 1 (setting $? to 1) if the file contains any Windows-style line endings, and with a status of 0 if it doesn't, making it useful in a shell if statement (note the lack of [ brackets ]):

if awk '/\r$/ { exit(1) }' filename ; then
    echo filename has Unix-style line endings
else
    echo filename has at least one Windows-style line ending
fi

A file can contain a mixture of Unix-style and Windows-style line endings. I'm assuming here that you want to detect files that have any Windows-style line endings.

Answer 10

I have been using

cat -v filename.txt | diff - filename.txt

which seems to work. I find the output a little easier to read than

dos2unix < filename.txt | diff - filename.txt

It is also useful if you can't install dos2unix for some reason.

Answer 11

file tells you the line endings, but only if they are not Unix-style:

❯ echo "hello1\nhello2\n" > hello-unix.txt
❯ cp hello-unix.txt hello-dos.txt
❯ cp hello-unix.txt hello-mac.txt
❯ unix2dos hello-dos.txt
unix2dos: converting file hello-dos.txt to DOS format...
❯ unix2mac hello-mac.txt
unix2mac: converting file hello-mac.txt to Mac format...
❯ file hello-unix.txt
hello-unix.txt: ASCII text
❯ file hello-dos.txt
hello-dos.txt: ASCII text, with CRLF line terminators
❯ file hello-mac.txt
hello-mac.txt: ASCII text, with CR line terminators

So:

• If file reports "CRLF line terminators", the file is DOS-style
• If file reports "CR line terminators", the file is Mac-style
• If file doesn't mention line terminators, the file is Unix-style