I'm trying to learn how the $LANG variable behaves with gnome-terminal (and its character encoding preference option). I've been using iso8859-1 (latin1) as my main character-set and all my filenames are encoded as such.

For the following tests I'll do an ls -l of a directory with Spanish accented characters in their filenames:

Case #1:

  • gnome-terminal configured for ISO-8859-1
  • LANG set to "en_US-iso8859-1"
  • Result: I see all files correctly

Case #2:

  • gnome-terminal configured for UTF-8
  • LANG set to "en_US-iso8859-1"
  • Result: I see garbage characters for all spanish characters. This is expected as I changed the character-encoding for the terminal

Case #3:

  • gnome-terminal configured for ISO-8859-1
  • LANG set to "en_US-UTF-8"
  • Result: I see garbage characters for all spanish characters.

Why is that in this last case I see garbled characters? Shouldn't the output of ls send the filenames straight to gnome-terminal as they are? And since gnome-terminal is configured for ISO-8859-1, I would have expected them to look right.

For a moment I thought that, perhaps, maybe bash is considering my $LANG variable and performing some conversion. Then I switched my terminal to UTF-8 but I still can't see the characters right. I even piped the output of ls to xxd and to my surprise I still see the files encoded as they are: ISO-8859-1.

To wrap up: If my listing contains ISO-8859-1 characters and my terminal emulator is configured for the same character-encoding: Who's doing the conversion when LANG is set otherwise?

Thanks for any help you can provide.


3 Answers 3


Your setting for LANG must match the terminal's. More precisely, your setting for LC_CTYPE (the character encoding) must match the terminal's encoding, the other locale settings don't need to match. And the terminal's encoding is usually specified by an option of the terminal emulator and not by a locale variable. The LC_CTYPE combines two indications: it tells applications what encoding to use on the terminal (both for input and output), and it tells applications what encoding to use with files. In cases 2 and 3, you've told ls to display output in an encoding that's different from the terminal's, so the output is garbled.

If you work with both UTF-8 and latin-1 encodings at different times, configure your terminal to use UTF-8. This should cause it to set LC_CTYPE to a value indicating UTF-8; don't override this setting. (If the terminal emulator doesn't set LC_CTYPE, do override it in your shell startup file or for your whole session.) To work with latin-1 data in an UTF-8 terminal, use luit (included in the X utility suite).

LC_CTYPE=en_US.iso88591 luit

(You can use any other locale with the same encoding, e.g. LC_CTYPE=es_ES.iso88591 luit.)

  • Thanks Gilles for that wonderful explanation, especially for explaining the two indications for LC_CTYPE.
    – Craconia
    Commented Sep 21, 2012 at 2:06
  • Going back to my last case: I thought that, since all filenames were encoded in latin1 plus the fact that my final output device, the one creating the glyphs (my terminal) was also configured for latin1, I was expecting to see the files correctly (regardless of LC_CTYPE)...
    – Craconia
    Commented Sep 21, 2012 at 2:13
  • It never occurred to me that ls would consider LC_CTYPE (set to UTF-8 in this case) and would perform some kind of character-set validation: whenever it sees something not compatible with the character set it would spit a specific character (e.g."?"). I said "validation" because it won't perform a "conversion" as luit does. Is it like this?
    – Craconia
    Commented Sep 21, 2012 at 2:14
  • @Craconia In the third case, ls replaces the unprintable characters by ?. Most strings encoded in latin-1 that represent real words are have unprintable characters if interpreted as UTF-8. Commented Sep 21, 2012 at 8:19

In case #2 and #3 you're mixing two different encoding UTF-8 and Latin-1. In case #1 you're using Latin-1 for both, so you don't have a problem.

The ls command (and all other well behaving programms) use the LANG setting for determining the encoding.

You may mixing two different Languages, but you shouldn't mix two different encodings.

Ensure that the LC_* environment variables are also use the the same encoding as your LANG variable.

As a rule of thumb you should configure your system nowadays to use only UTF-8.

If you have to edit old fashioned data files (e.g. java properties) you should either use a specialized editor (e.g. java ide) or ensure the encoding with tools like iconv or `recode..

  • Thanks. Yes, I have plans to switch to UTF-8 in the near future. Got a bunch of filenames to convert plus many many text files. iconv & convmv to the rescue...
    – Craconia
    Commented Sep 21, 2012 at 2:03

This might be outside of your need, but ....

It turns out in RHEL5, and probably prior, many of the man pages have somehow for some gd foresaken reason, been ascii-ized. That is, the raw man page has been converted from its native character set to 7-bit ASCII. No matter what you do with LC and LANG, the man page for latin1 produces a man page which is effectively useless. All special (8-bit) characters within have been replaced with 7-bit placeholders (usually ??). I find this hilarious.

But the utf8 version of these man pages might exist in the language-specific directory. The trick is to ask for them by their right name. For instance, latin1 is actually iso_8859-1. If you do a man page on it, and your LANG settings are correct, you see what you expect; the man page is found in the language-specific subdir (en/man7/iso_8859-1.7). But if you ask for iso-8859-1, for some reason, you get the ASCII version.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .