Which terminal encodings are default on Linux, and which are most common?

Question

I need to make a decision regarding whether a complicated commercial program that I work on should assume a particular terminal encoding for Linux, or instead read it from the terminal (and if so, how).

It's pretty easy to guess which system and terminal encodings are most common on Windows. We can assume that most users configure these through the Control Panel, and that, for instance, their terminal encoding, which is usually non-Unicode, can be easily predicted from the standard configuration for that language/country. (For instance, on a US English machine, it will be OEM-437, while on a Russian machine, it will be OEM-866.)

But it's not clear to me how most users configure their system and terminal encodings on Linux. The savvy ones who often need to use non-ASCII characters probably use a UTF-8 encoding. But what proportion of Linux users fall into that category?

Nor is it clear which method most users use to configure their locale: changing the LANG environment variable, or something else.

A related question would be how Linux configures these by default. My own Linux machine at work (actually a virtual Debian 5 machine that runs via VMWare Player on my Windows machine) is set up by default to use a US-ASCII terminal encoding. However, I'm not sure whether that was set up by administrators at my workplace or that's the setting out of the box.

Please understand that I'm not looking for answers to "Which encoding do you personally use?" but rather some means by which I could figure out the distribution of encodings that Linux users are likely to be using.

Answer 1

The oldest character encoding used in consoles like VT52 was ASCII.

That basic decision has been carried over for many years. Most consoles use ASCII as the most basic character set as defined by ANSI. The next set of encodings (in the west) are the ISO-8859 sets (from 1 to 15). One for each language (language group). Being the most common the ISO-8859-1 (English), and the other in proportion to the corresponding language in use.

Then, the most general list of world characters is Unicode, which, in Linux, is usually encoded in UTF-8.

It is that encoding the most common for present day terminals and programs in Linux.

---

From more general to particular settings:

OS

The default in debian since Etch on Apr 8th 2007 (13 years ago) has been utf-8.

Note : Fresh Debian/Etch installation have UTF8 enabled by default.

And confirmed on the release notes:

The default encoding for new Debian GNU/Linux installations is UTF-8. A number of applications will also be set up to use UTF-8 by default.

What that means is that Debian (and Ubuntu, Mint, and many other) are utf-8 capable by default.

locale

Which encoding (and country) is actually chosen by the user with the command dpkg-reconfigure locales is left to user preferences.

That configure the actual particular setting for the computer locale command.

All of the LC_* "environment variables" have specific effects on each of country/language sections (parts) as defined by the POSIX spec.

tty

But the above are just "general" settings. A particular terminal may (or may not) match it. Well, in general, the usual encoding for most terminals today is utf8.

The encoding for a particular terminal (tty) may be found if set to utf8 with:

$ stty -a | grep -o '.iutf8'
 iutf8

That is, no - before the result printed.

terminal

But the terminal (GUI window) inside which the tty terminal is (usually) running also has its own locale setting. If the settings are sane, probably:

$ locale charmap
UTF-8

Will have the correct answer.

But that is just a quick and very shallow look at all the i18n settings of linux/unix.

Take away: Probably, assuming Linux is using utf8 is your best bet.

Answer 2

I would use a similar heuristic you are using with Windows users, but via the LANG environmental variable. For example, on my system:

$ echo $LANG
en_US.UTF-8

Here, the code is saying I am using the English language, but with UTF-8 encoding of filenames and files.

As a general rule, Linux users using UTF-8 will have "UTF-8" at the end of their LANG environmental variable.

Answer 3

Modern Linux installations (for at least some 5 years, probably longer) use UTF-8. How that is handled by setting the environment values LC_CTYPE, LANG, and LANGUAGE. See for example the discussions here or here (Unicode centered).

Answer 4

For reasonably modern Linux/Unix systems, you shouldn't need to worry about terminal encoding. Just use getwchar or fgetws to read from stdin (or the terminal). [Note 1]

As man getwchar says, in the Notes section:

It is reasonable to expect that getwchar() will actually read a multibyte sequence from standard input and then convert it to a wide character.

There is a similar note in man fgetws.

With Linux, it is also reasonable to expect the encoding of wchar_t to be unicode, regardless of locale. The C99 standard allows the implementation to define the macro __STDC_ISO_10646__ to indicate that wchar_t values correspond to Unicode code points [Note 2], so you can insert a compile-time check for this expectation, which should succeed on modern Linux installs with standard toolchains. It's likely to succeed on modern Unix systems as well, although there is no guarantee.

---

Notes:

[1] You do need to initialize the locale by calling setlocale(LC_ALL, ""); once at the beginning of program execution. See man setlocale.

[2] The value of __STDC_ISO_10646__ is a date (in format yyyymmL) corresponding to the date of the applicable version of the Unicode standard. The precise wording from the standard (draft) is:

The following macro names are conditionally defined by the implementation:

__STDC_ISO_10646__ An integer constant of the form yyyymmL (for example, 199712L). If this symbol is defined, then every character in the Unicode required set, when stored in an object of type wchar_t, has the same value as the short identifier of that character. The Unicode required set consists of all the characters that are defined by ISO/IEC 10646, along with all amendments and technical corrigenda, as of the specified year and month. If some other encoding is used, the macro shall not be defined and the actual encoding used is implementation-defined.