1

I have tried every method I could find in the forums, and cannot display any 8-bit character in an xterm window, under kubuntu 18.04 (nor in earlier versions). All characters in range 0x20-0x7e display as expected, but none in range 0x80-0xfe. When I try, depending on the settings, I get either a blank or the default glyph of question mark in a dark oval. My simple test is:

echo -e '\xa2 \xa3'

These are characters 162 & 163(decimal) and in western fonts should display as cent sign and British pound. I've tried a selection of characters above 128 (= 0x80), same result for all. The different adjustments I've tested:

  • Set locale to a UTF-8 flavor.

  • Set to a UTF-8 encoding (eg. en_US.UTF-8).

  • Start xterm with different fonts invoked, all of which have the full 128-255 set of characters.

  • Tried uxterm and xterm.

  • In addition to the simple echo -e test, used test programs that display full font grids or invoke appropriate vt-100 esc-command sequences and strings. For instance:

    escape ( '<' (to load the DEC Supplemental Character Set into G1)

    ctrl-N (shift-out, to load G1 into the "left-half" GL set)

    \x32 \x33

In all cases, only the default '?' glyphs are displayed.

Plenty of other people have written to the forums with similar problems, and they were solved by adjustments as in the above list. None worked for me.

I'm running 32-bit kubuntu, rather than 64-bit. Could this be a factor in the problem?

We have a custom program that invokes an xterm-based editor, using curses tools, which displays at least one character in the 128-255 range. That character appeared fine running under Sun Solaris, but displays a blank in kubuntu linux with ncurses. Restoring that glyph is what put me on this chase.

I'd be grateful for any help, and happy to provide any and all detail.

0

Your shell's locale settings are part of the problem:

  • Set locale to a UTF-8 flavor.

  • Set to a UTF-8 encoding (eg. en_US.UTF-8).

That tells applications running inside xterm to use UTF-8. UTF-8 encoding uses codes in the 0x80-0xff range to build up multi-byte characters, which is not what you want.

The locale in effect when you start xterm affects how it interprets the same codes. If that locale tells xterm that it uses UTF-8, xterm will use UTF-8 encoding (see the locale resource), and depending on the resource settings, may not allow you to turn it off. (This is particularly an issue when running xterm in a desktop environment, where your system locale uses UTF-8, e.g., en_US.UTF-8). You can see what xterm is doing using the control-right-mouse menu: there is an entry "UTF-8 Encoding" which is checked when it expects UTF-8, and grayed-out when you cannot change it.

If your shell-initialization uses the system's locale settings, then it should be enough to do this from the command-line:

LC_ALL=en_US LANG=en-US xterm

Rather than UTF-8, what you appear to be asking about are the ISO-8859-1 and related encodings. Those are what the locale names without the ".UTF-8" suffix usually refer to.

Here is a screenshot from vttest illustrating ISO-8859-1 (what you might expect to see, for the application you are attempting to use):

vttest with ISO-8859-1

and this is what it would show with UTF-8 encoding

vttest with UTF-8 encoding

The ncurses library checks the locale (which the calling application should have initialized) and finds that those single bytes in the 0x80-0xff do not form complete multibyte UTF-8, and shows blanks. But if your locale settings (and terminal) are set consistently, you will see the expected characters.


On the other hand, your question mentions DEC Supplemental. That is different, because it would rely upon xterm's Unicode support (to use all of the available characters in the National Replacement Character Sets). Latin-1 is mapped 1-1 into Unicode, but DEC Supplemental (which is much like Latin-1) is not.

NRCS (National Replacement Character Sets) is provided as a mode in xterm. (The original hardware terminals used a setup selection). If your application actually uses DEC Supplemental (which is much like Latin-1) you might see something like this (with vttest highlighting the places where it does not match Latin-1):

vttest with DEC Supplemental

or it might use DEC Supplemental Graphic (again, similar):

vttest with DEC Supplemental Graphic

Either way, xterm can do that (with UTF-8 enabled). However, with the very old versions distributed by Ubuntu, you may have to compile your own programs. But in the context of the question, it seems that you are actually using Latin-1 rather than these older pre-standard character sets.

  • Thank you @Thomas Dickey for clearing up my confusion about using a UTF-8 - enabled xterm. Indeed. Setting the locale and LANG of xterm to the basic en_US solved my problem. Single-byte characters in the x80-xff range display correctly with that setting. Your detailed and illustrated answer was extremely helpful in explaining how it all fits together. Just as have your answers to so many other posters on this forum. – PRButterfield Jul 16 '19 at 21:38
  • As far as the DEC Supplemental character set, l got that name from the DEC VT-100 manual, where it's presented as one of the basic sets. I assumed it was still in general use, but no. So it's Latin-1 we are using. – PRButterfield Jul 16 '19 at 21:48
  • I think the terminology changed over time, and have to check my summary when I want to get the terms correct, anyway. – Thomas Dickey Jul 16 '19 at 22:22
0

There are different encodings in use for non-ASCII characters. The older ISO-8859-x encodings use one byte per character. The characters in your example, cent and pound, are encoded using octets 0xa2 and 0xa3 in the ISO-8859-1 (Latin1) character set. UTF-8 uses a variable length scheme, where the cent character is encoded as the two octet sequence 0xc2 0xa2, and the pound sing as 0xc2 0xa3.

To display the characters correctly, you need to set the locale to match the character encoding used in the text you want to display. You will either have to set the locale to ISO-8859-1, or recode your text files to UTF-8.

The advantage of UTF-8 over the ISO-8859-x encodings is that UTF-8 encompasses the whole Unicode range, whereas the older 8-bit encodings only cover 192 visible characters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.