5

I have 2 similar files (dos.txt and unix.txt) with text The quick brown fox jumps\n over the lazy dog. that differ by line endings. When I search for a word at the end of line, output from dos.txt is empty:

$ grep -E 'jumps^M?$' dos.txt unix.txt

unix.txt:The quick brown fox jumps

Grep finds something but doesn't print it. Actual output from grep looks like this:

$ grep -E --color=always 'jumps^M?$' dos.txt unix.txt | cat -v
^[[35m^[[Kdos.txt^[[m^[[K^[[36m^[[K:^[[m^[[KThe ... ^[[01;31m^[[Kjumps^M^[[m^[[K
^[[35m^[[Kunix.txt^[[m^[[K^[[36m^[[K:^[[m^[[KThe ... ^[[01;31m^[[Kjumps^[[m^[[K

So it looks like the only difference is that ^M is inside colored output and it causes whole line to disappear. How can I fix this (without converting dos files using dos2unix or similar tools)?

  • ^M? does not work for me. Is there something special about it? – Oskar Skog Mar 9 '17 at 19:40
  • 1
    @OskarSkog: You have to input hex 0x0d. In Bash you can do this by preceding ctrl-m by ctrl-v – Thor Mar 9 '17 at 19:50
  • I can confirm this behavior. It looks like a bug. – Thor Mar 9 '17 at 19:52
  • 1
    @Thor: grep doesn't know any better than to echo the carriage return at the end of the line, and the terminal does what it is told to. It's a question of should unix/GNU tools convert DOS line endings to unix line endings or not before processing files. Sometimes it would sense, other times it wouldn't. – Oskar Skog Mar 9 '17 at 20:01
  • 1
    @Thor: Tried with echo, ^M truncates. I also just realized that grep was expecting ^M and obviously wants to confirm it was found. But it seems that GNU grep generally understands DOS line endings. – Oskar Skog Mar 9 '17 at 21:36
8

After some searching for ^[[K escape sequence, reading half of a book about VT100 terminal and checking man grep I have found that setting environment variable GREP_COLORS to

GREP_COLORS=ne

Gives desired output:

$ export GREP_COLORS=ne
$ grep -E --color=always 'jumps^M?$' dos.txt unix.txt
dos.txt:The quick brown fox jumps
unix.txt:The quick brown fox jumps

$ grep -E --color=always 'jumps^M?$' dos.txt unix.txt | cat -v
^[[35mdos.txt^[[m^[[36m:^[[mThe ... ^[[01;31mjumps^M^[[m
^[[35munix.txt^[[m^[[36m:^[[mThe ... ^[[01;31mjumps^[[m

From grep man page:

ne     Boolean  value  that prevents clearing to the
       end of line using Erase in Line (EL) to Right
       (\33[K)  each  time  a  colorized  item ends.
       This is needed on terminals on  which  EL  is
       not  supported.   It  is  otherwise useful on
       terminals  for  which  the   back_color_erase
       (bce)  boolean  terminfo  capability does not
       apply, when the chosen  highlight  colors  do
       not  affect the background, or when EL is too
       slow or causes too much flicker.  The default
       is false (i.e., the capability is omitted).

In my case it works good even if I set highlight color to something that change background:

export GREP_COLORS="ne:mt=41;38"

Now the interesting question is why ^[[K produces this blank line.

Character ^M means carriage return without going to next line:

$ echo -e "start^Mend"
endrt

^[[K clears line from cursor to right and then writes rest of line:

$ echo -e "start\033[Kend"
startend

However when you put ^M before ^[[K it removes content:

$ echo -e "start^M\033[Kend"
end

After writing start cursor goes to the beginning of line then ^[[K removes everything and rest of the line is written. In case of grep output first line writes everything up to word jumps, then goes back to beginning of line ^M, writes harmless ^[[m sequence and ^J that goes to new line. This is why ^[[K after ^M clears whole line.

1

You can pipe the output through tr(1) and drop the carriage returns. tr -d '\r' will drop all carriage returns.

This works:

grep --color=always -E 'jumps' dos.txt unix.txt | tr -d '\r'

(Also works with the ^M?.)

Tip: Pipe output to less(1) whenever you suspect that the output may contain unsafe characters.

  • The questioner actually put xyr finger right on what was clearing the rest of the line, in xyr own answer. It is not the linefeed. – JdeBP Mar 9 '17 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.