Prevent tail from modifying the charset of the terminal

Question

In a Linux terminal (CentOS) I am using the command tail --follow=name my-rolling-file.log in order to see the logs of my application.

Sometimes in the log, there is some binary data dumped (I dump the body of a Camel Message that usually contains strings, but sometimes binary and/or special characters like Chinese in UTF-8) and when it happens my terminal gets corrupted like the pipe characters | are now ö instead.

I just guessed it is the binary data logging who can cause the problem and I am wondering if it is possible to ask the tail command to ignore special characters. I checked the man page but did not find anything there.

Currently to fix the problem I have to Ctrl-C the tailing, make a reset in the terminal and relaunch the tail command. I want to prevent these operations if possible.

If you know another command than tail, but with the same features (following rolling files) it would be acceptable as well as long it is installable and runnable under CentOS 6.5.

Answer 1

You can translate special characters (binary data) into ordinary characters that are safe to display by piping your tail command into cat -v:

tail --follow=name my-rolling-file.log | cat -v

The -v (verbose) option of cat (also known as --show-nonprinting) displays "nonprinting" characters using the ^ and M- notations:

• 0x00 is displayed as ^@
• 0x01 is displayed as ^A
• 0x02 is displayed as ^B
• etc.
• 0x7F is displayed as ^?
• 0x80-0xFF are displayed as M- followed by the representation of the low order seven bits of the byte.

Or, if you prefer to discard the characters, use:

tail --follow=name my-rolling-file.log | tr -cd "\t\n -~"

tr "translates" characters; i.e., maps them.  It can be used, for example, to change lower case to upper case.  The -d option tells it to delete characters; for example, tr -d aeiou deletes all (lower case) vowels.  The -c option means complement; i.e., do all characters other than the ones I specify.  "\t\n -~" is tab, newline, and all the ordinary printable characters.

Answer 2

What about this,

tail --follow=name my-rolling-file.log | strings

The default for strings is that it will only output printable characters in lengths of 4 (or more), but you can change this with -n {number}.

Answer 3

Convert (some troublesome) characters to . with tr:

tail -f data | tr "\000-\011\013-\037\177-\377" "."

Answer 4

Let less handle the problem, as it's good at it.

The pager less is pretty good at formating binary files for output. Also, the pager features, like scrollback and searching are often useful.
It has no direct option for following a file that grows, but it has way to provide keystrokes to "type" after startup.

The key F starts following the growing file, updating the screen when the file grows.
Prefix the key with + instead of - and use that just like an option:

Replacing tail -f:

less +F growing-file.log

To continue following the file by name, even if it's moved away or deleted, and recreated with the same name, add the option --follow-name; It modifies what the key F and +F do, it does not start following by itself:

Replacing tail -F (--follow=name):

less --follow-name +F rolling-file.log

If the file is recognized as a "binary file", less asks whether you really want to see it - to avoid the question, use the option -f (--force) too:

less --follow-name +F -f rolling-binary-file.log

(Note that the meaning of -f for less is different from the same option for tail.)

Use Ctrl+c to stop following, which allows you to scroll back or search.

• Start following again with F.
• Jump to start: g.
• Jump to start: G.
• Search: /.
• Quit: q.

Binary output example

$ less +F /usr/bin/vim 
"/usr/bin/vim" may be a binary file.  See it anyway?

After pressing y (which you can avoid with -f) the screen shows:

 ^A^@^@^@^@^@<E5>
 ^A^@^@^@^@^@<E6>
 ^A^@^@^@^@^@^@^P^A^@^@^@^@^@^B^P^A^@^@^@^@^@8^P^A^@^@^@^@^@F^P^A^@^@^@^@^@
 <F1>^A^@^@^@^@^@^P<F1>^A^@^@^@^@^@-<F1>^A^@^@^@^@^@0<F1>^A^@^@^@^@^@i<F1>^A
 ^L(^@^@^@^@^@^@^@^@^@^@^@^@^@<A3><A5>&^@^@^@^@^@^@^@^@^@^@^@^@^@b['^@^@^@^@
 @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@<A7>&^@^@^@^@
 <AB>&^@^@^@^@^@Ȣ&^@^@^@^@^@^L,^@^@^@^@^@^@<FF><FF><FF><FF><FF><FF><FF><FF>
  ^@^@^@^@^@^@s]&^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^_<D3>%^@^@^@^@^@̢
 <AF>&^@^@^@^@^@<CA><(^@^@^@^@^@^B^D^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
 <B1>&^@^@^@^@^@^D^D^H^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^R<B1>&^@^@
 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<A9><B3>&^@^@^@^@^@D<B8>&^@^@
 <B5>&^@^@^@^@^@^]T&^@^@^@^@^@^BL^@^@^@^@^@^@H<8A>O^@^@^@^@^@F@^@^@^@^@^@^@
 K'^@^@^@^@^@^C<C4><FF><FF>^@^@^@^@^OK'^@^@^@^@^@^C<C3><FF><FF>^@^@^@^@^TK'
 #^@^@^@^@^@^D^@^@^@^@^@^@^@<E7>v&^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
 #^@^@^@^@^@^D^@^@^@^@^@^@^@<E7>v&^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
 #^@^@^@^@^@p|#^@^@^@^@^@P~#^@^@^@^@^@^@^@^@^@^@^@^@^@^P
 #^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<90>{#^@^@^@^@^@^@^@^@^@^@^@^@^@
Waiting for data... (interrupt to abort)

Control chars are shown in carret notation like ^A (^@ is the null byte).
They are also highlighted in the terminal (which is not shown above), to make the character Control-A, ^A different from the character ^ followed by A.

Configuration of how to show binary characters

From man less:

Control  and  binary  characters  are  displayed  in  standout (reverse
video).  Each such character is displayed in caret notation if possible
(e.g.  ^A for control-A).  Caret notation is used only if inverting the
0100 bit results in a normal printable character.  Otherwise, the char‐
acter  is displayed as a hex number in angle brackets.  This format can
be changed by setting the LESSBINFMT environment variable.   LESSBINFMT
may begin with a "*" and one character to select the display attribute:
"*k" is blinking, "*d" is bold, "*u" is underlined, "*s"  is  standout,
and  "*n"  is  normal.  If LESSBINFMT does not begin with a "*", normal
attribute is assumed.  The remainder of LESSBINFMT is  a  string  which
may  include one printf-style escape sequence (a % followed by x, X, o,
d, etc.).  For example, if LESSBINFMT is  "*u[%x]",  binary  characters
are  displayed  in  underlined hexadecimal surrounded by brackets.  The
default if no LESSBINFMT is  specified  is  "*s<%02X>".   Warning:  the
result  of  expanding the character via LESSBINFMT must be less than 31
characters.

When the character set is utf-8, the LESSUTFBINFMT environment variable
acts similarly to LESSBINFMT but it applies to Unicode code points that
were successfully decoded but are unsuitable for display  (e.g.,  unas‐
signed  code  points).   Its  default  value is "<U+%04lX>".  Note that
LESSUTFBINFMT and LESSBINFMT  share  their  display  attribute  setting
("*x")  so specifying one will affect both; LESSUTFBINFMT is read after
LESSBINFMT so its setting, if any,  will  have  priority.   Problematic
octets  in  a  UTF-8  file (octets of a truncated sequence, octets of a
complete but non-shortest form  sequence,  illegal  octets,  and  stray
trailing  octets)  are displayed individually using LESSBINFMT so as to
facilitate diagnostic of how the UTF-8 file is ill-formed.