3

I need to make a decision regarding whether a complicated commercial program that I work on should assume a particular terminal encoding for Linux, or instead read it from the terminal (and if so, how).

It's pretty easy to guess which system and terminal encodings are most common on Windows. We can assume that most users configure these through the Control Panel, and that, for instance, their terminal encoding, which is usually non-Unicode, can be easily predicted from the standard configuration for that language/country. (For instance, on a US English machine, it will be OEM-437, while on a Russian machine, it will be OEM-866.)

But it's not clear to me how most users configure their system and terminal encodings on Linux. The savvy ones who often need to use non-ASCII characters probably use a UTF-8 encoding. But what proportion of Linux users fall into that category?

Nor is it clear which method most users use to configure their locale: changing the LANG environment variable, or something else.

A related question would be how Linux configures these by default. My own Linux machine at work (actually a virtual Debian 5 machine that runs via VMWare Player on my Windows machine) is set up by default to use a US-ASCII terminal encoding. However, I'm not sure whether that was set up by administrators at my workplace or that's the setting out of the box.

Please understand that I'm not looking for answers to "Which encoding do you personally use?" but rather some means by which I could figure out the distribution of encodings that Linux users are likely to be using.

1

From more general to particular settings:

OS

The default in debian since Etch on Apr 8th 2007 (11 years ago) has been utf-8.

Note : Fresh Debian/Etch installation have UTF8 enabled by default.

And confirmed on the release notes:

The default encoding for new Debian GNU/Linux installations is UTF-8. A number of applications will also be set up to use UTF-8 by default.

What that means is that debian (and ubuntu, mint, and many other) are utf-8 capable by default.

locale

Which encoding (and country) is actually chosen by the user with the command dpkg-reconfigure locales is left to user preferences.

That configure the actual particular setting for the computer locale command:

$ locale
LANG=en_US.utf8
LANGUAGE=
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

All of those "environment variables" have specific effects on each of country/language sections (parts) as defined by the POSIX spec.

tty

But the above are just "general" settings. A particular terminal may (or may not) match it. Well, in general, the usual encoding for most terminals today is utf8.

The encoding for a particular terminal (tty) may be found if set to utf8 with:

$ stty -a | grep -o '.iutf8'
 iutf8

That is, no - before the result printed.

terminal

But the terminal (GUI window) inside which the tty terminal is (usually) running also has its own locale setting. If the settings are sane, probably:

$ locale charmap
UTF-8

Will have the correct answer.

But that is just a quick and very shallow look at all the i18n settings of linux/unix.

Take away: Probably, assuming linux is using utf8 is your best bet.

2

I would use a similar heuristic you are using with Windows users, but via the LANG environmental variable. For example, on my system:

$ echo $LANG
en_US.UTF-8

Here, the code is saying I am using the English language, but with UTF-8 encoding of filenames and files.

As a general rule, Linux users using UTF-8 will have "UTF-8" at the end of their LANG environmental variable.

1

Modern Linux installations (for at least some 5 years, probably longer) use UTF-8. How that is handled by setting the environment values LC_CTYPE, LANG, and LANGUAGE. See for example the discussions here or here (Unicode centered).

0

For reasonably modern Linux/Unix systems, you shouldn't need to worry about terminal encoding. Just use getwchar or fgetws to read from stdin (or the terminal). [Note 1]

As man getwchar says, in the Notes section:

It is reasonable to expect that getwchar() will actually read a multibyte sequence from standard input and then convert it to a wide character.

There is a similar note in man fgetws.

With Linux, it is also reasonable to expect the encoding of wchar_t to be unicode, regardless of locale. The C99 standard allows the implementation to define the macro __STDC_ISO_10646__ to indicate that wchar_t values correspond to Unicode code points [Note 2], so you can insert a compile-time check for this expectation, which should succeed on modern Linux installs with standard toolchains. It's likely to succeed on modern Unix systems as well, although there is no guarantee.


Notes:

[1] You do need to initialize the locale by calling setlocale(LC_ALL, ""); once at the beginning of program execution. See man setlocale.

[2] The value of __STDC_ISO_10646__ is a date (in format yyyymmL) corresponding to the date of the applicable version of the Unicode standard. The precise wording from the standard (draft) is:

The following macro names are conditionally defined by the implementation:

__STDC_ISO_10646__ An integer constant of the form yyyymmL (for example, 199712L). If this symbol is defined, then every character in the Unicode required set, when stored in an object of type wchar_t, has the same value as the short identifier of that character. The Unicode required set consists of all the characters that are defined by ISO/IEC 10646, along with all amendments and technical corrigenda, as of the specified year and month. If some other encoding is used, the macro shall not be defined and the actual encoding used is implementation-defined.

  • Trivia: the date of the macro actually corresponds to the ISO-10646 standard, 199712L corresponds to a non-compatible change, where Korean hangul was moved from some block to another (the "Korean mess", alluded to in the UTF-8 RFC). – ninjalj Jun 12 '16 at 10:34
  • "it is also reasonable to expect the encoding of wchar_t to be unicode" - Unicode is not an encoding, it's the thing being encoded. Do you mean UTF-16-be, UTF-16-le, UCS-2, ... – Eric Dec 9 '18 at 23:27
  • @Eric: "Unicode provides a unique number for every character." (What is Unicode). It is an encoding and the standard quote in my answer uses precisely that word. And here's another one, the first answer in the Unicode FAQ: "Unicode is the universal character encoding, maintained by the Unicode Consortium." In other words, Unicode encodes every (well, many) characters into a Unicode codepoint... – rici Dec 10 '18 at 1:17
  • 1
    UTF-16 is a "Unicode Encoding Form" (Section 3.9) which represents each Unicode codepoint as a series of possibly smaller numbers. UTF-16BE is one of the "Encoding Schemes" (Section 3.10) used to turn a UTF-16 sequence into a series of bytes for the purposes of interchange between applications. These distinctions involve a lot of hair-splitting, but the precise language can be useful and is described at length in the Unicode book, which is freely available online... – rici Dec 10 '18 at 1:18
  • 1
    @eric: there never was a UCS-32; you might be thinking of UCS-4, a term which is no longer used. ISO-10646 defines UCS-4 as the UTF-32 encoding form, which represents Unicode code points in 32-bit binary numbers. Unicode is just abstract mapping (ir encoding) of characters to numbers, without regard to the representation of the numbers. Hiwever, since the range of possible Unicode values is [0, 0x10FFFF], all Unicode codepoints can be represented in 32 bits by zero extending. In that sense, yes, there is a correspondence. – rici Dec 10 '18 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.