Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to learn how the $LANG variable behaves with gnome-terminal (and its character encoding preference option). I've been using iso8859-1 (latin1) as my main character-set and all my filenames are encoded as such.

For the following tests I'll do an ls -l of a directory with Spanish accented characters in their filenames:

Case #1:

  • gnome-terminal configured for ISO-8859-1
  • LANG set to "en_US-iso8859-1"
  • Result: I see all files correctly

Case #2:

  • gnome-terminal configured for UTF-8
  • LANG set to "en_US-iso8859-1"
  • Result: I see garbage characters for all spanish characters. This is expected as I changed the character-encoding for the terminal

Case #3:

  • gnome-terminal configured for ISO-8859-1
  • LANG set to "en_US-UTF-8"
  • Result: I see garbage characters for all spanish characters.

Why is that in this last case I see garbled characters? Shouldn't the output of ls send the filenames straight to gnome-terminal as they are? And since gnome-terminal is configured for ISO-8859-1, I would have expected them to look right.

For a moment I thought that, perhaps, maybe bash is considering my $LANG variable and performing some conversion. Then I switched my terminal to UTF-8 but I still can't see the characters right. I even piped the output of ls to xxd and to my surprise I still see the files encoded as they are: ISO-8859-1.

To wrap up: If my listing contains ISO-8859-1 characters and my terminal emulator is configured for the same character-encoding: Who's doing the conversion when LANG is set otherwise?

Thanks for any help you can provide.

Craconia

share|improve this question

2 Answers 2

up vote 0 down vote accepted

Your setting for LANG must match the terminal's. More precisely, your setting for LC_CTYPE (the character encoding) must match the terminal's encoding, the other locale settings don't need to match. And the terminal's encoding is usually specified by an option of the terminal emulator and not by a locale variable. The LC_CTYPE combines two indications: it tells applications what encoding to use on the terminal (both for input and output), and it tells applications what encoding to use with files. In cases 2 and 3, you've told ls to display output in an encoding that's different from the terminal's, so the output is garbled.

If you work with both UTF-8 and latin-1 encodings at different times, configure your terminal to use UTF-8. This should cause it to set LC_CTYPE to a value indicating UTF-8; don't override this setting. (If the terminal emulator doesn't set LC_CTYPE, do override it in your shell startup file or for your whole session.) To work with latin-1 data in an UTF-8 terminal, use luit (included in the X utility suite).

LC_CTYPE=en_US.iso88591 luit

(You can use any other locale with the same encoding, e.g. LC_CTYPE=es_ES.iso88591 luit.)

share|improve this answer
    
Thanks Gilles for that wonderful explanation, especially for explaining the two indications for LC_CTYPE. –  Craconia Sep 21 '12 at 2:06
    
Going back to my last case: I thought that, since all filenames were encoded in latin1 plus the fact that my final output device, the one creating the glyphs (my terminal) was also configured for latin1, I was expecting to see the files correctly (regardless of LC_CTYPE)... –  Craconia Sep 21 '12 at 2:13
    
It never occurred to me that ls would consider LC_CTYPE (set to UTF-8 in this case) and would perform some kind of character-set validation: whenever it sees something not compatible with the character set it would spit a specific character (e.g."?"). I said "validation" because it won't perform a "conversion" as luit does. Is it like this? –  Craconia Sep 21 '12 at 2:14
    
@Craconia In the third case, ls replaces the unprintable characters by ?. Most strings encoded in latin-1 that represent real words are have unprintable characters if interpreted as UTF-8. –  Gilles Sep 21 '12 at 8:19

In case #2 and #3 you're mixing two different encoding UTF-8 and Latin-1. In case #1 you're using Latin-1 for both, so you don't have a problem.

The ls command (and all other well behaving programms) use the LANG setting for determining the encoding.

You may mixing two different Languages, but you shouldn't mix two different encodings.

Ensure that the LC_* environment variables are also use the the same encoding as your LANG variable.

As a rule of thumb you should configure your system nowadays to use only UTF-8.

If you have to edit old fashioned data files (e.g. java properties) you should either use a specialized editor (e.g. java ide) or ensure the encoding with tools like iconv or `recode..

share|improve this answer
    
Thanks. Yes, I have plans to switch to UTF-8 in the near future. Got a bunch of filenames to convert plus many many text files. iconv & convmv to the rescue... –  Craconia Sep 21 '12 at 2:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.