sv_SE.UTF-8 Problems with Bash and VIM in iTerm2

Question

Two specific issues, despite me running iconv -f iso-8859-15 on my old files when converting from sv_SE.iso8859-15 to sv_SE.utf-8 I still get distorted characters. And I believe it's because of Vim because I can't even type Swedish åäö characters in Vim any longer.

The second issue is in the Bash, when open a new shell, or start a new line, the first Swedish åäö I type does not show, until I press any second key that produces any character.

So typing å/ on my keyboard will only show å in the terminal. Typing åäö will show åö, and so forth. It always ignores the second character I type after one of these, and then starts showing at the third. Sort of like ignoring the second byte or skipping the second byte in the multibyte character.

Edit: Clarification, after I've typed the first multibyte character, others show directly. But if I type åååå four times, the first one is delayed, and then hit enter, and then typ exit what comes out is åexit. As if it saves the last byte of the last character I typed in buffer.

Here's the relevant environment configuration, please let me know if there is more needed to solve this.

$ cat .inputrc
set show-all-if-ambiguous on
set mark-directories on
set mark-symlinked-directories on
set page-completions off
set visible-stats on
set completion-query-items 9001

set input-meta on
set output-meta on
set convert-meta off

"\e[A": history-search-backward
"\e[B": history-search-forward
"\C-w": backward-kill-word

"\e[1~": beginning-of-line
"\e[4~": end-of-line
"\e[3~": delete-char

$ locale
LANG="sv_SE.UTF-8"
LC_COLLATE="sv_SE.UTF-8"
LC_CTYPE="sv_SE.UTF-8"
LC_MESSAGES="sv_SE.UTF-8"
LC_MONETARY="sv_SE.UTF-8"
LC_NUMERIC="sv_SE.UTF-8"
LC_TIME="sv_SE.UTF-8"
LC_ALL="sv_SE.UTF-8"

$ locale -a
sv_SE
sv_SE.ISO8859-1
sv_SE.ISO8859-15
sv_SE.UTF-8

Also tried this in .inputrc, advised from a Swedish mac forum. Did not help with any of the above issues. Same symptoms in both Terminal.app and iTerm2.app, with the checkbox "Escape non-ASCII characters" unchecked in Terminal.app.

set meta-flag on
set input-meta off
set output-meta on
set convert-meta off

In Vim.

• Current language: "sv_SE.UTF-8/sv_SE.UTF-8/sv_SE.UTF-8/C/sv_SE.UTF-8/sv_SE.UTF-8"
• set encoding=UTF-8
• set fileencoding=UTF-8

I'm running the latest update of Mac OS Lion with the latest beta of iTerm2. Same issue in Terminal.app so I expect this to be caused by my environment.

Answer 1

As far as the terminal issues go, I'd try using a terminal emulator that is known to have mature Unicode support, like uxterm. It helps if the locale-related environment variables are made available to the shell executable upon startup, i.e. not set in your ~/.profile or similar shell startup script.

I don't know if this will help you, but I fixed my Vim-related Unicode woes with the following addition to my ~/.vimrc:

" Unicode options
if has("multi_byte")
    " set the display encoding
    " (default is "", or "utf-8" in the GUI)
    if &termencoding == ""
        " we're probably not using the GUI
        " note: :set won't allow &-replacement
        let &termencoding = &encoding
    endif
    " set the internal encoding
    set encoding=utf-8

    " &fileencoding (controls how characters in the internal encoding will
    " be written to the file) will be set according to &fileencodings
    " (default: "ucs-bom", "ucs-bom,utf-8,default,latin1" when 'encoding'
    "  is set to a Unicode value)
endif " has("multi_byte")

Answer 2

Changing a locale from non- to a UTF-8 based one requires more than just environment settings. It requires used applications to be built with wide character support. The symptoms you describe point at lacks of such support - this might be an issue with libraries such as slang or ncurses for example.

Try to look for something like "UTF-8 migration guide" for OSX, that should guide you how to replace some of your software packages with versions that are wide-character-enabled. It may turn out, that some of applications you currently use (like the terminal app) just don't have options of supporting this. But this is not much likely, since UTF has become more and more widely used.