Terminal Character Encoding Issue BitchX

Question

I ran BitchX on three different Linux OS-es because I got this weird � (question mark) on two of them (Linux OpenSUSE 11.4 and BackTrack 5 R1) using yakuake/konsole (OpenSUSE) and terminator/tilda (BT 5 R1).

However, on PuppyLinux Lucid 528 with RoxTerm the character is displayed as it should be (it is the Greek character "fi"). Now, on both SUSE and BT I have checked to see the value of $LC_CTYPE, and it's the same as in Puppy Linux (namely en_US, so no utf-8 encoding).

I've experimented on both SUSE and BT with different character encodings (from terminal's settings), I've noticed the changes, but the text was still sometimes garbled, instead of the question mark I got some weird unintelligible "u"-s (in Puppy it is always displayed correctly).

What could cause this annoying problem?
What could be different and how can I check the difference?

Answer 1

There are several things going on here.

Can you first make sure that LC_CTYPE=en_US does mean "no utf-8"?

The nl_langinfo function can help, for example:

$ python
>>> import locale
>>> locale.setlocale(locale.LC_CTYPE, 'en_US')
'en_US'
>>> locale.nl_langinfo(locale.CODESET)
'ISO-8859-1'

Secondly, it would be helpful to try changing only one variable at a time. For example, if RoxTerm works in PuppyLinux, does it work in OpenSUSE and/or BackTrack? And can you also say which version of BitchX you're using, in case it makes a difference?

Finally, IRC doesn't really supply any character encoding information, so the server and all the users in a channel have to agree on a character encoding. I think these days most users use UTF-8, so maybe you should try changing your settings to UTF-8?

---

UPDATE

You said BitchX doesn't support UTF-8.

Then I think you are going to have problems talking to other users.

But if everyone else is talking Greek, and also using BitchX, then perhaps you want to use the Greek ISO-8859-7 character set.

On my system, the only two locales that support ISO-8859-7 out of the box are Greek language ones, e.g. el_GR.iso88597.

Run locale -a to see a full list of your options, and choose one that includes that character set.

Answer 2

Question mark in rhomb likely means that your client program accepts character and passes it to displaying engine (am I correct that bitchX is only terminal program?), but terminal fails to render it. So it is likely terminal configuration problem - e.g. the selected font doesn't contain glyphs for Greek characters. You can test it by copy&paste from this terminal to another one with something like "hexdump -C" running - it would show phi codepoint (U+03C6, so `CF 86' in hexdump of utf-8). If so, try to select font which supports Greek.

Seeing something like \u03C6, , etc. means likely locale is broken (not Unicode) so client accepts characters, apply Unicode treating to them but then fails to output them directly and switches to fallback mode. So, I guess SuSE and BT setups both have en_US as alias to non-Unicode coding, likely ISO-8859-1. Try to set en_US.UTF-8 directly and compare effects.