Is there a safe cat?

Question

From time to time it happens that I cat a binary either from curl or from the local filesystem. In most cases the broken terminal can be fixed with reset. In other cases, particularly if the binary is large, the terminal will be stuck for several minutes printing output like this:

aka

c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;
2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;
2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;
2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;2c1;

I have three questions regarding this scenario;

1. What does 2c1 mean and why is the terminal printing this?
2. Have you seen a cat in the wild, guarding against this undesired behavior in an interactive session?
3. Do you have any suggestions on how to program such a cat (in cee or golang)

My initial instinct was to wrap cat in a function to detect this, but I soon realized that it is fairly difficult to get right and would have numerous edge-cases.

function cat() {
    # warn user if
    #   - argument 1 is a large  executable 
    #   - argument 1 to the previous command in the a pipe-chain looks like a large binary
    # abort if
    #   - session is interactive and we are able to detect 2c1 garbage
}

---

A practical solution could be to always use less (with LESSPIPE) when looking at "unsafe" input, but this question is not about pagers. I am aware of less and lesspipe. I use them actively every day. Perhaps less+lesspipe is the solution to this problem, that the author(s) of less implemented some 20-30 year ago facing the same issue.

However, cat is different from a "pager" in more than one way... Primarily cat is non-interactive. This is significant to me.

The suggestion about less+lesspipe is genuenly good (imho) in practical terms, but I am more concerned with the nitty-gritty of control characters, special escape sequences and how different terminals handle these inputs.

I am more interested in the technical nitty-gritty details of control characters and how terminals or shells interpret "garbage" and control characters. I am not asking "how would you solve this problem". I am asking "why is the terminal handling binary files like this".

Answer 1

I would use less instead, which warns of binary files and on some systems can handle various kinds (e.g. on CentOS 7, I can do less file.rpm and see the files in the RPM). I believe that's called "lesspipe."

Also, next time that happens, you can try reset or tput reset to get back to normal. Those send escape sequences to the terminal that tell it to reset to a default sane state and also do the equivalent of stty sane which changes the settings of the tty device file to a sane default (dumping a binary file should not affect those though). reset can also fix the tty device file's notion of the window size of the terminal or terminal emulator (as reported by tty size) with those terminals that support querying it.

Answer 2

Nobody has mentioned 'strings'. Although strings isn't exactly like cat, it does print just the text strings from continuous text data to make it safer to view in the terminal. It usually comes with the binutils package. It is a handy program for quickly ensuring that you don't get any binary output from printed data and is also useful if you just want to see the continuous non-binary. Note that by default it only prints out continuous text sections with 4 or more ASCII characters. This can be adjusted with the -n option.

Answer 3

You interact with a terminal or terminal emulator through a serial line or pseudo-tty device (which emulates a serial line).

Though there is a software module in the kernel that sits in the middle as a kind of adaption layer and does some transformation (discussed briefly later), you typically:

• send a stream of bytes to the terminal over that serial line which the terminal interprets either as glyphs to render on its screen or special instructions to change its behaviour
• in return, the terminal sends a stream of bytes to the computer over another wire on that serial line to tell the computer what you typed or to respond to some of the control queries it received.

For instance, a terminal could be configured as a ISO8859-1 (aka latin1 terminal) which means that when it receives the 0x53 0x74 0xe9 0x70 0x68 0x61 0x6e 0x65, it interprets it as rendering the S, t, é, p, h, a, n, e glyphs at the current cursor position on its screen. And conversely, when the user types S, the terminal sends the 0x53 byte.

Byte values in the range 0 to 0x1f are interpreted as control characters. That is, they are not represented as glyphs, but have special meaning.

For instance:

• 0x7 (BEL) generate an audio or video alert
• 0x8 (BS) moves the cursor to the left
• 0xa (LF) moves the cursor down
• 0xd (CR) moves the cursor to the first column of the screen
• 0x9 (TAB) moves the cursor to the next tabulation

There are only 32 control characters in that range and most terminals have many more features or way you can control them. So beside those, you can send sequences of more than one byte to control your terminal. For most terminals and for most of those sequences, the first byte is 0x1b (ESC) followed by one or more bytes.

For instance, while there are control characters to move the cursor left or down as seen above, there is none to move it to the right or up (as originally, in tele-typewriters, right would be done with "space", but in CRT terminals that erases what's under the cursor, and you wouldnt go up with a tele-typewriter as that would likely create a paper jam), so escape sequences had to be introduced for those, on most terminals 0x1b 0x5b 0x43 and 0x1b 0x5b 0x41 respectively (incidently, that's also the byte sequence many terminals send upon pressing the Right and Up for those that have such keys).

Now among the escape sequences that terminals support are some that:

• change the text or background colour and other graphic rendering attributes
• change the charset. For instance, there's no Greek character in latin1, and terminals (from the pre-Unicode days and still today) support switching to a different charset to display letters of other languages, or box drawing characters.
• set the position of tab stops
• can query information from the terminal such as the cursor position, colour, window title, size...
• can affect how input is processed. For instance, some terminals support entering a mode whereby upon pressing Shift+A for instance, it doesn't send a 0x41 (ASCII A) character but a sequence of bytes that encodes information about modifiers (shift, alt, ctrl...) and keycode.
• some X11 terminal emulators recognise escape sequences to change the font, the window size, display JPEG images, send the screen contents to a printer...

In a text file, you usually only have bytes (or byte sequences if UTF-8 or other multibyte charsets) representing graphical characters. The only control characters you'll find in text files are NL (0xa, aka LF) and TAB (0x9).

When you do cat file.txt, cat just reads the contents of file.txt and writes it to its stdout. If stdout is a serial or pseudo-tty device file (/dev/ttyS0, /dev/pts/0 for instance) that has a terminal line discipline pushed onto it as would be the case if you run that command from an interactive shell in a terminal emulator, the line discipline translates those NLs to CR+NL (though NLNL may be translated to just CRNLNL) so the terminal upon receiving CRNL will move the cursor to the start and then down.

So the text in the contents of the file will be displayed on the terminal screen provided the text in the file is encoded in the character set of the terminal.

Now, bytes in executable files or other random binary files are not intended to represent characters, they can have any value including ones in the range 0 to 31, so when sent to a terminal, the terminal will do what it's told and interpret them as control characters, which may make it do anything as listed above and much more and render it completely unusable.

To guard against that, first you don't send those files to a terminal as that wouldn't make sense, or if you don't know whether a file may be a text file (or a file intended to be viewed verbatim by a terminal with escape sequences intended to be interpreted by a terminal) or not, you can use a tool that either removes the control characters (at least all those but TAB and NL) or give them a visual graphical representation.

That's what the -v and -t options as supported by many cat implementations do. Where with -v, all but NL and TAB are converted to some ^X notation for bytes 0 to 31 and 0x7f, M-^X for bytes 0x80 to 0x9f and 0xff and M-X for bytes 0xa0 to 0xfe which are common visual representations of non-ASCII characters. And -t does it just for TAB (changed to ^I).

Or you can use a pager such as less or vim's view which do that by default (at least as long as you don't use the -r/-R raw options) and are a bit smarter in that they don't transform non-ASCII characters that are meant to have graphical representations in your locale and make it clearer what bytes have been transformed by using colouring or standout modes.

Or you can use tools dedicated to previewing non-text files such as hexdump -C or xxd.

See also the l command of sed which does something similar to cat -vte and is standard (contrary to cat -vte) in a less ambiguous way:

sed -n l < a-file

Answer 4

• Yes, young padawan.

$ cargo search bat 
bat = "0.23.0"            # A cat(1) clone with wings.

• But master, what is a cargo?

$ cargo --help |any install rust 
Rust's package manager
      --list                List installed commands
      --explain <CODE>      Run `rustc --explain CODE`
    install     Install a Rust binary. Default location is $HOME/.cargo/bin
    uninstall   Uninstall a Rust binary
$ cargo install bat
(...)

master: This happens if you try and bat a binary.

$ bat `which bat`
[bat warning]: Binary content from file '/home/jaroslav/.cargo/bin/bat'
will not be printed to the terminal (but will be present if the output
of 'bat' is piped). You can use 'bat -A' to show the binary file contents.

• padawan: Are there any downsides to bat, master?
• master: Yes, it can be slow on large files, but that is partially because it does highlighting of structured syntax. That stuff can be disabled e.g. --style=plain --color=never
• padawan: What about the weird characters that cat outputs to the terminal then, master?
• master: That happens because terminals will happily accept and interpret anything that looks like an ANSI escape code (internal terminal command) and try to do what the command says, if that command is implemented. For a brief intro, check out this list of ansi color escape sequences

Here is how to reproduce this behavior:

$ echo -ne "\u1B\u5B\u63" | xxd
00000000: 1b5b 63                                  .[c

$ echo -ne "\u1B\u5B\u63" 
^[[?1;2c

$ 1;2c