UTF-8 characters are not displayed correctly in Debian

Question

Short description of my problem:
I ran into an issue lately where I am unable to make bash/nano/irssi/etc display "special" UTF-8 characters like the german umlauts (äüö), the euro sign (€) and some other UTF-8 characters like ß, §, etc.

What I already tried:

• dpkg-reconfigure locales and only generated en_US.UTF-8
• setting LC_ALL, LANG and LANGUAGE to en_US.UTF-8 within the .bashrc for both my user and root
• re-installed locales and libx11-data (which seems to contain all the language data)

Of course I re-logged in via ssh after all these changes and even tried restarting the server even though I know it doesn't solve any problem in Linux in 99,9875% of all cases.

Information on my system:
OS: Debian stretch -> Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2 x86_64 GNU/Linux
locales: v.2.22-7

Output of locale:

LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

When typing for example ä into the console and press enter I get -bash: $'\344': command not found.
Honestly I am out of ideas, can anyone help me out with this?

Answer 1

You've told bash and other applications that your terminal uses the UTF-8 encoding. That's good only if your terminal actually does use UTF-8. Bash doesn't get to decide that the terminal encoding is, the terminal gets to decide.

If you want to use UTF-8, configure your terminal to use UTF-8. Since you're using SSH, you need to configure whatever terminal you're running the SSH client in to use UTF-8. That's the default on most modern systems, but apparently yours isn't set up this way.

You should avoid setting LC_CTYPE explicitly in a terminal: ideally the terminal will set this. However this doesn't always work, especially over SSH (on many systems, the SSH server forbids the client from setting LC_CTYPE).

If you need to set the environment variable, the right place would be .profile, not .bashrc.

Answer 2

It sounds as if you are using the Linux console (rather than one of the X-based terminal emulators), and that it is not running in UTF-8 mode. I would use this script to turn it on (and investigate to see why it is off):

#!/bin/sh
# send character-string to enable UTF-8 mode
if test ".$1" = ".off" ; then
        printf '\033%%@'
else
        printf '\033%%G'
fi

that is, call the script utf8, and type

utf8 on

To investigate the error messages, I made a script like this, in two flavors (one in UTF-8, and the other in ISO-8859-1):

#!/bin/bash
printf "ä\n"
echo "ä"
ä

The UTF-8 script says

$ ./foo
ä
ä
./foo: line 4: ä: command not found

and the ISO-8859-1 script says (in a terminal using a locale with UTF-8 encoding):

$ ./foo2
�
�
./foo2: line 5: $'\344': command not found

The point is that bash adjusts its error message to correspond to the locale, and seeing that it cannot show the ISO-8859-1 character in the UTF-8 locale, shows it as an octal number.