tr command unable to process colour output piped from grep

Question

I'm running macOS 12.3.1

I added a couple of lines to my .zshrc, viz.

export GREP_OPTIONS='--color=always'
export GREP_COLOR='1;35;40'

After this, when I pipe grep output to tr, it returns the same number of lines but all of the lines are blank

for example:

grep ^.....$ /usr/share/dict/words | tr "[:lower:]" "[:upper:]"

Returns 10230 blank lines.

Is this expected?

Answer 1

To output matches in colour, grep writes colouring escape sequences before and after the match.

Those are instructions to the terminals to change their background and/or foreground colour.

It's important to realise that it is in the output along with the text. You don't see it because your terminal doesn't display them as graphical symbols but instead understand it as special instructions.

The escape sequences start with an ESC character (0x1b byte (033 in octal) in ASCII aka \e or ^[) and are followed by a few characters which themselves don't have to be control characters.

You can reveal those characters by piping the output to things like:

$ echo + | grep --color=always . | sed -n l
\033[01;31m\033[K+\033[m\033[K$

$ echo + | grep --color=always . | od -An -vtc -tx1 -to1
 033   [   0   1   ;   3   1   m 033   [   K   + 033   [   m 033
  1b  5b  30  31  3b  33  31  6d  1b  5b  4b  2b  1b  5b  6d  1b
 033 133 060 061 073 063 061 155 033 133 113 053 033 133 155 033
   [   K  \n
  5b  4b  0a
 133 113 012

(here also including hexadecimal and octal values of the individual bytes)

Or (though non-standard and ambiguous):

$ echo + | grep --color=always . | cat -A
^[[01;31m^[[K+^[[m^[[K$

You can see that in the grep output, there's a \e[01;31\e[K before the match and \e[m\e[K after the match.

What escape sequences a given terminal recognises and how varies with the terminal. For xterm for instance, see the specification there. Those above these days are rather ubiquitous.

For the one that starts with \e[ and ends in m, the terminal understands each of the ;-separated numbers as different rendering attributes to apply to text that is going to be written from now on. For instance 1 is for bold, 31 sets the foreground colour to red.

\e[K is the escape sequence that tells the terminal to clear the screen from the cursor to the end of the line.

So the terminal actually sees:

<bold-fg_red><clear-to-eol>+<reset-all-attributes><clear-to-eol>

But all tr sees is those ESC, [, ... m along with the other ones that it's been asked to transliterate.

In particular here, it will transliterate m to M, and the escape sequence that was changing colour attributes will turn into something else together.

To find out about escape sequences and what they do, other than looking at the terminal documentation (such as https://www.invisible-island.net/xterm/ctlseqs/ctlseqs.html mentioned above for xterm), which is sometimes hard to find or inexistent, you can also look in the terminfo database which records the escape sequences recognised by a number of terminals for a few common actions.

You can query that database by hand for your terminal (identified by the $TERM environment variable) with infocmp:

$ infocmp -xL1 | grep M,
        delete_line=\E[M,
        key_enter=\EOM,
        key_mouse=\E[M,
        parm_delete_line=\E[%p1%dM,
        scroll_reverse=\EM,

And for the details of what those actions (capabilities) are, you can look at the terminfo(5) man page (man 5 terminfo).

\e[M (delete_line) deletes one line, \e[<decimal>M (parm_delete_line) deletes <decimal> lines. So your colouring sequences have turned into line deleting sequences once transliterated to upper case.

You generally don't want to post-process coloured output as those are only intended for terminals. That's why most commands that support colouring disable it when their output doesn't go to a terminal.

For GNU grep, as you already found out, you need --color=auto (or grep --color) to get that behaviour.

Now, if you do still want to see colours, you need to move the colouring to the last command in the pipeline, the one that has its output go to the terminal:

<file tr '[:lower:]' '[:upper:]' | grep  -xE --colour=auto '.{5}'

Here using --colour=auto so that if ever the script that contains that command has its output redirected / post-processed, the colouring is disabled.

Here, since the regexp matches the whole line (-x option above, which avoids having to use ^ and $ like in your approach), you might as well switch the foreground colour to red before and clear attributes after:

if [ -t 1 ]; then
  tput setaf 1 # set ANSI foreground colour
  tput bold
fi
grep...
if [ -t 1 ]; then
  tput sgr0 # turn off all attributes
fi

Here using tput to query terminfo for the right sequence for your terminal, though since most terminals do it the same, do like grep and hardcode the sequences:

[ -t 1 ] && printf '\33[1;31m'
grep...
[ -t 1 ] && printf '\e[m'

Using [ -t 1 ] to check that stdout (file descriptor 1) is a terminal.

Answer 2

I fixed it by changing GREP_OPTIONS from --color=always to --color=auto

I suppose tr and colour coded text don't go well together