Strange Konsole character encoding behavior

Question

I set the character encoding of Konsole in Lubuntu to UTF-16 (not sure if Konsole uses UTF-16 little endian or UTF-16 big endian).

Then I run the following command in the Run window:

Then I pressed the "a" button on my keyboard (the keyboard layout selected in Lubuntu is English), which I assume will cause the bytes 61 00 (if Konsole uses UTF-16 little endian) to be sent to the line discipline, and the line discipline will in turn echo these bytes back to Konsole, and Konsole will display the character "a". But instead I got the following:

Again I pressed the "a" button and I got the following:

Also again I pressed the "a" button and I got the following:

Also again I pressed the "a" button and I got the following:

Why am I getting these strange characters instead of simply getting the characters "aaaa"?

---

Edit:

These are the characters displayed in Konsole:

幡｀懾䁞幡｀懾䁞

Answer 1

Let's even further emphasize that xxd -p is irrelevant, we're not talking about its output. It hasn't even seen the input due to line buffering on the kernel's side, and hence hasn't produced any output. For what it's worth, it could also be a cat or a sleep 100000 or whatever. We're talking about how the kernel (the line discipline) echoes back the input.

If you switch back to UTF-8 and then press an Enter to xxd -p, its output goes like fffe6100fffe6100. So little endian is confirmed (or probably it's the architecture's native byte order), but surprisingly, there's a BOM before every character. Makes me suspect that it wasn't properly thought through by Konsole developers, they just blindly invoke iconv with UTF-16 (without BE or LE specified) as the target character set, for each chunk of input as it becomes available, and iconv places it there.

Let's strace konsole to see what it does on its file descriptor connected to /dev/ptmx:

write(..., "\377\376a\0", 4)             = 4
[...]
read(..., "\377\376a^@", 5)              = 5

The NUL byte (0x00) comes back as literal ^@, that is, 0x5e followed by 0x40.

Along with a (0x61), this gives you U+5e61, which is exactly the first glyph you see appearing. Plus, you'll be off by a byte, that is, the next supposedly low byte will be interpreted as high byte, and vice versa.

For the byte 0x00 the kernel simply mangles how it's echoed back. For some other bytes it performs other actions as well. For example, the byte 0x03 (^C) typically triggers an interrupt to be sent to the foreground process, 0x15 (^U) wipes out the data you've entered so far, 0x0a and/or 0x0d (i.e. newline) flushes the data to your app, etc. All these bytes can (and do) legally occur within the UTF-16 representation of characters, and you sure don't want any of these happening while typing your input.

In order to use UTF-16 on the line disciple, the kernel would need to provide explicit support for this and would need to be told that this encoding is being used (something along the lines of an stty utf16). To my best knowledge, this is not implemented (luckily – it would be a total waste of developer resources). The kernel expects an ASCII-compatible encoding to be used, which UTF-16 isn't.

Even if UTF-16 was implemented in the kernel for the tty line, the entire ecosystem would necessarily be quite fragile. The terminal can receive data from multiple sources simultaneously, and there's no way to guarantee that all producers of data and all transporters (e.g. ssh) can always keep the bytes coupled in pairs. Once it goes off by a byte (as seen above), the rest is unusable.

I'm now even more certain that Konsole developer didn't properly think about it. In my opinion UTF-16 should be removed from their list of offered encodings, or at least a warning should be shown. I've submitted Konsole bug 395171.