Encoding Problem: Qt Apps delete all non ASCII chars from files

Question

Qt applications are deleting non-latin characters from ISO-8859 encoded files on my Gentoo system. Actually I'm trying to merge two German files with KDiff3 and P4Merge (making "Whlen" out of "Wählen"). Both tools don't display the Umlauts and when the file is saved, they also disappear in the file. The Dejavu Monospace font is used, Courier New shows the same behavior.

If UTF-8 encoded files are presented to the tools, all non-latins are handled correctly.

GTK Meld (and all other GTK apps) handles the chars (ISO-8859-1 and UTF-8) quite well. I believe its my Locale configuration, but can't discern what's amiss...

Any ideas?

Configs:

 $ locale -a
 C
 POSIX
 de_DE
 de_DE.iso88591
 de_DE.iso885915@euro
 de_DE.utf8
 de_DE@euro
 deutsch
 en_US
 en_US.iso88591
 en_US.utf8
 german

 $ locale
 LANG=en_US.UTF-8
 LC_CTYPE="en_US.UTF-8"
 LC_NUMERIC="en_US.UTF-8"
 LC_TIME="en_US.UTF-8"
 LC_COLLATE=C
 LC_MONETARY="en_US.UTF-8"
 LC_MESSAGES="en_US.UTF-8"
 LC_PAPER="en_US.UTF-8"
 LC_NAME="en_US.UTF-8"
 LC_ADDRESS="en_US.UTF-8"
 LC_TELEPHONE="en_US.UTF-8"
 LC_MEASUREMENT="en_US.UTF-8"
 LC_IDENTIFICATION="en_US.UTF-8"
 LC_ALL=

Qt use flags for x11-libs/qt-core-4.8.2:4:

exceptions glib iconv qt3support ssl (-aqua) -c++0x -debug -icu -pch
(-optimized-qmake%) (-qpa%)

Answer 1

If UTF-8 encoded files are presented to the tools, all non-latins are handled correctly.

Then I suggest you recode the files to UTF-8, merge them, then recode back if neccessary.

GTK Meld (and all other GTK apps) handles the chars (ISO-8859-1 and UTF-8) quite well.

I guess you might call this a feature, not a bug: when the application sees some byte sequences it cannot interpret as characters in the current locale, it has several options. It might start guessing, which apparently is what GTK does. Or it might consider the corresponding bytes to be errors and drop them from the file. Or it might inform the user about these issues, which would be even nicer, but apparently KDE doesn't go this far.

I believe its my Locale configuration, but can't discern what's amiss...

Your locale indicates that text files are expected to be encoded as UTF-8. Your files however are not. So there is a discrepancy. You can fix that by either recoding the files, or by starting the process in question with a different locale. The LC_CTYPE facet should be the relevant one, but start with LC_ALL first just to be sure. When invoking the command from the command line, simply prepend the correct setting to the command, i.e.

LC_CTYPE=en_US.iso88591 kdiff3 foo bar