How to find a locale name for a given character encoding?

Question

The man page for setlocale seems to say that the language code and the character encoding name are sufficient to set the appropriate locale:

A locale name is typically of the form language[_territory][.codeset][@modifier], where language is an ISO 639 language code, territory is an ISO 3166 country code, and codeset is a character set or encoding identifier like ISO-8859-1 or UTF-8.

However, a quick test shows that only the "modifier" part of a locale name is optional:

void tryLocale(const char * locid)
{
    char * result = std::setlocale(LC_CTYPE, locid);
    std::cout << locid << " = " << (result ? result : "fail") << std::endl;
}

int main()
{
    tryLocale("de");           // de = fail
    tryLocale("de_DE");        // de_DE = fail
    tryLocale("de_DE.CP1252"); // de_DE.CP1252 = de_DE.CP1252
    tryLocale("de.CP1252");    // de.CP1252 = fail
    tryLocale(".CP1252");      // .CP1252 = fail
}

My problem is that I only know the desired encoding name (e.g. ISO-8859-1) and I may come up with the language code (e.g. en). I have no idea how to find an appropriate country name (e.g. US), and I'm not interested in a country anyway: I just want functions like tolower in my app to use the correct codepage.

Answer 1

I think you'll have to loop through them. In zsh:

for l (${(f)"$(locale -a)"}) 
  [[ $(LC_ALL=$l locale charmap) = ISO-8859-1 ]] && print -r -- $l

Or the same using its $langinfo special associative array in the zsh/langinfo module:

zmodload zsh/langinfo
for LC_ALL (${(f)"$(locale -a)"})
  [[ $langinfo[CODESET] = ISO-8859-1 ]] && print -r -- $LC_ALL

Would list all the available locale that use ISO-8859-1 as the charmap.

But note that the LC_CTYPE category where the charmap / codeset is specified also covers character classification: what is a lower case letter, what is punctuation, etc and also transliterations (like that used by tolower()), both of which could vary from one region / country to the next even if the same codeset is used.

For instance, see how lowercase I is ı in GNU Turkish locales, regardless of what charmap is being used (UTF-8, ISO-8859-9...), while it's i in most other locales that also use UTF-8.

You can have a look at the locale source definitions, for instance with:

(cd /usr/share/i18n/locales && pcregrep -Me '(?ms)^LC_CTYPE.*?^END' -- *)

On a GNU system to see the differences across locales for the LC_CTYPE category. You won't find the charmap in there, locales for combinations of those files and charmaps are generated using localedef -i thosefiles -f charmap, though only some combinations make sense, see /usr/share/i18n/SUPPORTED for the list.

For instance, the en_GB locale on your system might have been generated with localedef -i locales/en_GB -f charmaps/ISO-8859-1.gz and the en_GB.UTF-8 one with localedef -i locales/en_GB -f charmaps/UTF-8.gz.

So here, maybe you need to find a locale that uses ISO-8859-1 as the charmap, but also with transliteration rules and character classifications that make sense in mainland Britain for British English or for German speakers in Italy / Germany etc, like for instance that satisfy:

[[ $(locale language)  = 'British English' &&
   $(locale territory) = 'United Kingdom' &&
   $(locale charmap)   = ISO-8859-1 ]]

Which should narrow the choice somewhat.

Note that language and territory are non-standard GNU extensions, which explains why you won't find them in zsh's $langinfo. The GNU libc documentation (info libc langinfo) only mentions:

The file ‘langinfo.h’ defines a lot more symbols but none of them are official. Using them is not portable, and the format of the return values might change. Therefore we recommended (sic) you not use them.

/usr/include/langinfo.h on my system has:

  _NL_IDENTIFICATION_LANGUAGE,
  _NL_IDENTIFICATION_TERRITORY,

See also locale -k LC_IDENTIFICATION, locale -k LC_CTYPE for the list of keywords supported for a given locale category on GNU systems (locale -kc LC_ALL used to work but no longer does these days apparently).