Weird symbols on screen when using grep?

Question

Any idea what could be causing this? Without using grep, the only stuff display there are the ISO codes and empty space.

Software used

Command: ./trans --id --input /path/to/txt | grep ISO | grep [a-z]

root@box /test # alias grep
alias grep='grep --color=auto'
root@box /test # type grep
grep is aliased to `grep --color=auto'

Normal output:

Answer 1

The screenshot appears to show mangled ANSI colour codes, which control text rendering. Bold/bright text is produced with the sequence ␛[1m, which is usually interpreted by your terminal and not displayed on the screen directly: it just makes the next bit of text bright. The screenshot of the ungrepped output shows that colour difference between the labels and values on each line, so the original output is using them.

It appears that that sequence has been broken by your final grep, which matched the "m" in the code (since it's a letter [a-z]) and tried to highlight it in red itself. That left a partial escape sequence behind, which your terminal couldn't process.

The escape character ␛ is U+001B, which is the hexadecimal number that's rendered in the unknown-character boxes. What's displayed is the escape (the box), a [, a 1, a red m followed by the expected matching text "eng", and the same happening at the end with "22" (the numeric code for "normal colour & intensity").

---

The broken output is really:

 ␛[1␛[31mmeng␛[22m␛[22␛[31mm␛[22m 

where ␛[31m makes text red and ␛[22m turns it back to white, both inserted by grep around the m characters into the original text. The original was just:

 ␛[1meng␛[22m 

which is just bright "eng" and then a switch back to normal text.

You could check this by changing your final grep into grep --color=always and piping into hexdump, which will show all the unprintable characters and the ones interpreted by your terminal.

---

You can deal with this a few ways. One is to use grep without your alias for the moment:

./trans --id --input /path/to/txt | grep ISO | \grep [a-z]

The backslash temporarily skips the alias and runs grep directly.

Another is to strip out the ANSI codes from the original command, for which there are some suggestions in this question:

./trans --id --input /path/to/txt | perl -pe 's/\e\[[\d;]*m//g' | grep ISO | grep [a-z]

Yet another option is to add an extraneous pipe on the end:

./trans --id --input /path/to/txt | grep ISO | grep [a-z] | cat

Because the final grep's output isn't directly to the TTY, but to cat via a pipe, it won't insert the coloured highlighting.

Perhaps the best option is to get Translate Shell to stop using terminal control sequences in its own output in the first place when it is not to a terminal. That would properly involve a bug report from you to its author(s) and a code fix to Translate Shell's ansi() function, but one can bodge it somewhat:

TERM=dumb ./trans --id --input /path/to/txt | grep ISO | grep [a-z]

This passes the dumb terminal type in Translate Shell's environment, which it at least recognizes as not having ECMA-48 colour support. (Sadly, Translate Shell does not use terminfo, and merely hardwires in its own code the terminal types that it understands and the control sequences that it uses.)