Why does 'cal' use weird 08 / ^H / \b terminal code for highlighting and how does that work?

Question

When you run cal on Linux, the output for the current month will reverse video highlight the current day. When I send that output to hexdump -c, I get some interesting results:

0000000               N   o   v   e   m   b   e   r       2   0   1   6
0000010                          \n   S   u       M   o       T   u    
0000020   W   e       T   h       F   r       S   a          \n        
0000030                       1           2       _  \b       _  \b   3
0000040           4           5          \n       6           7        
0000050   8           9       1   0       1   1       1   2          \n
0000060   1   3       1   4       1   5       1   6       1   7       1
0000070   8       1   9          \n   2   0       2   1       2   2    
0000080   2   3       2   4       2   5       2   6          \n   2   7
0000090       2   8       2   9       3   0                            
00000a0                  \n                                            
00000b0                                              \n                
00000bc

As you can see, there is an invisible sequence being printed of _\b _\b before the '3' that is highlighted for today. _ being underscore (5F in ascii hex) and \b being Ctrl-H or 08 in ASCII hex. What is this? I know there are a lot of obscure terminal codes, but I would expect it to use something more standard like \e[7m. What is even stranger is that I can't reproduce the same behavior of cal by printing out the same characters using standard printf functions like one of these commands:

/usr/bin/printf "1 2 _\b _\b3 4 5\n"
/usr/bin/printf "1 2 _^H _^H3 4 5\n"

Where ^H is made by pressing Ctrl-V Ctrl-H. But neither of these produce the same inverse video output that cal does. I even tried writing a little C program to do it too. I've also tried with echo -e. The interesting thing is that while it doesn't reverse the video in the terminal, if I pipe the output from less -R, it changes its color to yellow and underlines it. On other terminals I tried it just underlines it. It almost seems like overstriking, but if I use a character other than _ it doesn't work, which makes me think that _\b is a single code sequence. And how does the video for that character then get inversed?

Any insight into this?

The man page says that the output of cal is supposed to be a bit for bit compatible version to the original Unix cal command. So I can only assume this is some ancient code.

Answer 1

It almost seems like overstriking

That's exactly what it is.  As discussed at Why there are 11 tabstops on a 80-column console?, it helps to think of the action of mechanical typewriters when it comes to Unix terminals.  In this case, the sequence _ BS (the backspace character) before a character is a convention that is used to indicate underlining of that character, because on some terminals that's how one underlined text. An alternative control sequence is BS _ after the character. On the original terminals it didn't matter what overstruck what, of course.  On contemporary video terminals, the last character that is written “wins”, erasing previous data.  Therefore, the _ BS <character> order is preferred.

FreeBSD ncal, which is what this program is, has two modes of operation when it comes to highlighting.

• If its output is a terminal, it looks up the so and se sequences for the current terminal type in the termcap database and emits those either side of the highlighted text. (There's actually a bug in the code that does this, to do with a buffer on the stack going out of scope and its contents being used later, that no-one seems to have spotted.)
• If its output is not a terminal, it emits the text with each character to be highlighted preceded by a _ BS sequence.

You cannot replicate this by emitting a _ BS sequence to your terminal, unless (of course) your terminal is one of the terminals where this is how one underlines stuff. This isn't the case for terminal emulators, and almost certainly isn't the case for whatever terminal(s) or terminal emulator(s) you are using here.

You can, however, filter text that uses this convention, through the ul program, which recognizes this and several other typewriter-like conventions and translates them to whatever the control sequences for one's terminal actually are, looking them up in the termcap database. You can filter the outputs of your printf commands through ul, too.

On other terminals I tried it just underlines it.

Ironically, filtering the non-terminal-mode output of ncal through the ul program is in fact slightly superior to letting ncal write terminal control sequences itself. Whereas ncal uses the terminal's standout mode, ul will attempt to use the terminal's actual underline mode (if it has one) when translating the _ BS sequence. As the termcap manual explains, standout mode can be whatever is suitable to the terminal (including boldface, reverse video, or colours) and isn't necessarily underlining. On one of your terminals it is clearly a combination of underlining and a colour change.

Moreover, ul also copes with terminals that don't have underline start/end sequences, but that do have underline last character sequences. Ironically, ul will cope if your terminal really is one that underlines by having BS _ after each character, whereas ncal will not cope.

And of course ul doesn't have ncal's buffer handling bug. ☺

If I pipe the output into less -R, it changes its color to yellow and underlines it.

As you've spotted, the less program understands _ BS sequences and handles them somewhat as the ul program does. It's not completely the same. ul can handle sequences involving multiple _ and BS characters, and can also handle similar sequences for boldface. less cannot. Contrast what you see from these two:

• /usr/bin/printf "1 2 ______\b\b\b\b\b\b 3 _\b4. \b\b\b45 6\n" | ul

• /usr/bin/printf "1 2 ______\b\b\b\b\b\b 3 _\b4. \b\b\b45 6\n" | less

  
  

back in the good old days

Sadly, these still are "the good old days". Don't let people fool you into believing that this is rarely used nowadays.

It's not in the manual, but the source code for ul points out that it is attempting to implement the control sequence processing of a Teletype Model 37 because "that's what nroff outputs".  The GNU replacement for the original Unix nroff program, written long after terminals had gained fancy features like colours, boldface, and italics, is capable of generating ECMA-48 control sequences for colours, boldface, and italics. It actually does so in the normal case.

nroff, and its GNU replacement, are used for formatting manual pages for display on your terminal. Sadly, and ironically, starting about 10 years after it was written people proceeded to hobble the GNU tool so that it generated the old Teletype Model 37 sequences from 1968 instead of the "new" ECMA-48 control sequences from 1976 (sic!). They made man invoke groff with options that modified its default behaviour, and added undocumented files that forced extra ditroff output.

Every time that you read a manual page on your terminal, the manual system is running groff which is dutifully converting the manual source text into an output character stream with these old Teletype Model 37 control sequences, which less or more are converting into your terminal's control sequences.

Further reading

• Jonathan de Boyne Pollard (2016).  Italics and colour in manual pages on a nosh user-space virtual terminal (archived version, current version). The nosh package.
• Jonathan de Boyne Pollard (2017).  An improved manual page for ul (archived version, current version). Proposals.

Answer 2

Ctrl-H is backspace, it moves the cursor one step to the left. Sending an underscore, a backspace, and some other character was the way to underline something on a hard-copy ("paper") terminal back in the good old days. This was used to highlight the current day in the output of cal.

My cal program, when run in konsole does not output this sequence. If I run script -c cal and examine the resulting typescript file, I can see the cal program uses the escape sequence <esc>[7m to switch to inverse mode video.