45

I need to periodically run a command that ensures that some text files are kept in Linux mode. Unfortunately dos2unix always modifies the file, which would mess file's and folder's timestamps and cause unnecessary writes.

The script I write is in Bash, so I'd prefer answers based on Bash.

10 Answers 10

41

You can use dos2unix as a filter and compare its output to the original file:

dos2unix < myfile.txt | cmp -s - myfile.txt
  • 2
    Very smart and useful, because it tests the complete file and not only the first or a few line. – halloleo May 19 '15 at 0:04
  • 2
    Maybe you could replace test by myfile.txt twice in your example to avoid confusion with /usr/bin/test. – Peterino May 30 '16 at 7:51
  • 1
    NB you will need to delete the -s flag to see the output. From man pages: -s, --quiet, --silent suppress all normal output – tobalr Jan 29 '18 at 19:20
24

If the goal is just to avoid affecting the timestamp, dos2unix has a -k or --keepdate option which will keep the timestamp the same. It will still have to do a write to make the temporary file and rename it, but your timestamps will not be affected.

If any modification of the file is unacceptable, you can use the following solution from this answer.

find . -not -type d -exec file "{}" ";" | grep CRLF
  • 1
    Do you mean you literally write CRLF as 4 characters C, R, L and F? – bodacydo Dec 3 '15 at 3:19
  • 6
    Do you also mean that grep can take CR and LF just like that? – bodacydo Dec 3 '15 at 3:19
  • @bodacydo It's explained in the answer he links to, and now also in Scott's edit of BertS' answer here unix.stackexchange.com/a/79708/59699 . – dave_thompson_085 Dec 3 '15 at 5:14
  • @dave_thompson_085 I dont see explanation. It only mentions CRLF but doesnt explain what it is. – bodacydo Dec 3 '15 at 6:05
  • 1
    @bodacydo stackoverflow.com/questions/73833/… says that find ... -exec file ... | grep CRLF for a file with DOS line endings (i.e. bytes 0D 0A) "will get you something like: ./1/dos1.txt: ASCII text, with CRLF line terminators As you can see this contains the actual string CRLF and therefore is matched by grep looking for the simple string CRLF. – dave_thompson_085 Dec 4 '15 at 8:40
21

You could try to grep for CRLF code, octal:

grep -U $'\015' myfile.txt

or hex:

grep -U $'\x0D' myfile.txt
  • Of course, the assumption is that this is a text file. – mdpc Jun 17 '13 at 17:24
  • 1
    I like this grep usage because it allows me to easily list all such files in the directory with grep -lU $'\x0D' * and pass the output to xargs. – Melebius Apr 30 '15 at 6:19
  • whats the meaning of the $ before the search pattern? @don_crissti – fersarr Oct 31 '17 at 10:07
  • 1
    @fersarr - unix.stackexchange.com/a/401451/22142 – don_crissti Oct 31 '17 at 11:03
19

Since version 7.1 dos2unix has an -i, --info option to get information about line breaks. You can use dos2unix itself to test which files need conversion.

Example:

dos2unix -ic *.txt | xargs dos2unix
13

First method (grep):

Count the lines that contain a carriage return:

[[ $(grep -c $'\r' myfile.txt) -gt 0 ]] && echo dos

Count the lines that end with a carriage return:

[[ $(grep -c $'\r$' myfile.txt) -gt 0 ]] && echo dos

These will typically be equivalent; a carriage return in the interior of a line (i.e., not at the end) is rare.

More efficient:

grep -q $'\r' myfile.txt && echo dos

This is more efficient

  1. because it doesn't need to convert the count to an ASCII string, and then convert that string back to an integer, and compare it to zero, and
  2. because grep -c needs to read the entire file, to count all the occurrences of the pattern, while grep -q can exit upon seeing the first occurrence of the pattern.

Notes:

  • Throughout the above, you may need to add the -U option (i.e., use -cU or -qU), because GNU grep guesses whether the file is a text file.  If it thinks the file is text, it ignores carriage returns at the ends of lines, in an attempt to make $ in regular expressions work "correctly" — even if the regular expression is \r$!  Specifying -U (or --binary) overrules this guesswork, causing grep to treat the file(s) as binary and pass the data to the matching mechanism verbatim, with CR-endings intact.
  • Do not do grep … $'\r\n' myfile.txt, because grep treats \n as a pattern delimiter.  Just as grep -E 'foo|' looks for lines containing foo or a null string, grep $'\r\n' looks for lines containing \r or a null string, and every line matches a null string.

Second method (file):

[[ $(file myfile.txt) =~ CRLF ]] && echo dos

because file reports something like:

myfile.txt: UTF-8 Unicode text, with CRLF line terminators

Safer variant:

[[ $(file -b - < myfile.txt) =~ CRLF ]] && echo dos

where

  • file -b outputs only the file type, and not the file name.  Without this, a file whose name included the characters CRLF would trigger a false positive.
  • file - < filename works even if filename begins with -See Bash script: check if a file is a text file.

Beware that checking the output from file might not work in a non-English locale.

  • You can replace "$(echo -e '\r')" with the much simpler $'\r', although personally I'd use $'\r\n' to reduce the number of false positives. – rici Jun 17 '13 at 17:03
  • @rici grep $'\r\n' seems to match all files on my system... – depquid Jun 17 '13 at 17:09
  • @rici: good catch. I edited my answer according to your suggestion. — depquid: Maybe you are on Windows? :-) rici's tip works here. – BertS Jun 17 '13 at 17:11
  • @depquid (and BertS): Actually, I think the correct invocation is grep -U $'\r$', to prevent grep trying to second-guess line-endings. – rici Jun 17 '13 at 17:18
  • Also, you can use -q to just set the return code if a match is found, instead of -c which requires an additional check. Personally I like your second solution, although it's highly dependent on the whims of file and might not work in a non-English locale. – rici Jun 17 '13 at 17:22
8

Use cat -A

$ cat file
hello
hello

Now if this file was made in *NIX systems, it would display

$ cat -A file
hello$
hello$

But if this file was made in Windows, it would display

$ cat -A file
hello^M$
hello

^M represents CR and $ represents LF. Notice that Windows did not save the last line with CRLF

This does not change the file contents either.

  • The best and simplest solution! needs more up votes. – user648026 Jun 23 '18 at 16:34
4

a bash function for you:

# return 0 (true) if first line ends in CR
isDosFile() {
    [[ $(head -1 "$1") == *$'\r' ]]  
}

Then you can do stuff like

streamFile () {
    if isDosFile /tmp/foo.txt; then
        sed 's/\r$//' "$1"
    else
        cat "$1"
    fi
}

streamFile /tmp/foo.txt | process_lines_without_CR
  • 3
    You don't have to use isDosFile() in your example: streamFile() { sed 's/\r$//' "$1" ; }. – user26112 Jun 17 '13 at 18:59
  • 1
    I think this is the most elegant solution; it doesn't read whole file, just the first line. – Adam Ryczkowski Jun 22 '13 at 20:50
4

If a file has DOS/Windows-style CR-LF line endings, then if you look at it using a Unix-based tool you'll see CR ('\r') characters at the end of each line.

This command:

grep -l '^M$' filename

will print filename if the file contains one or more lines with Windows-style line endings, and will print nothing if it doesn't. Except that the ^M has to be a literal carriage return character, typically entered in the terminal by typing Ctrl+V followed by Enter (or Ctrl+V and then Ctrl+M). The bash shell lets you write a literal carriage return as $'\r' (documented here), so you can write:

grep -l $'\r$' filename

Other shells may provide a similar feature.

You can use another tool instead:

awk '/\r$/ { exit(1) }' filename

This will exit with a status of 1 (setting $? to 1) if the file contains any Windows-style line endings, and with a status of 0 if it doesn't, making it useful in a shell if statement (note the lack of [ brackets ]):

if awk '/\r$/ { exit(1) }' filename ; then
    echo filename has Unix-style line endings
else
    echo filename has at least one Windows-style line ending
fi

A file can contain a mixture of Unix-style and Windows-style line endings. I'm assuming here that you want to detect files that have any Windows-style line endings.

  • 1
    You can encode a carriage return on the command line in bash (and some other shells) by typing $'\r', as mentioned in other answers to this question. – Scott Dec 2 '15 at 6:41
2

Use file:

$ file README.md
README.md: ASCII text, with CRLF line terminators

$ dos2unix README.md
dos2unix: converting file README.md to Unix format...

$ file README.md
README.md: ASCII text
  • This idea has been discussed much more thoroughly in two previous answers. – G-Man May 10 '18 at 15:57
1

I have been using

cat -v filename.txt | diff - filename.txt

which seems to work. I find the output a little easier to read than

dos2unix < filename.txt | diff - filename.txt

It is also useful if you can't install dos2unix for some reason.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.