1

I set the character encoding of Konsole in Lubuntu to UTF-16 (not sure if Konsole uses UTF-16 little endian or UTF-16 big endian).

Then I run the following command in the Run window:

enter image description here

Then I pressed the "a" button on my keyboard (the keyboard layout selected in Lubuntu is English), which I assume will cause the bytes 61 00 (if Konsole uses UTF-16 little endian) to be sent to the line discipline, and the line discipline will in turn echo these bytes back to Konsole, and Konsole will display the character "a". But instead I got the following:

enter image description here

Again I pressed the "a" button and I got the following:

enter image description here

Also again I pressed the "a" button and I got the following:

enter image description here

Also again I pressed the "a" button and I got the following:

enter image description here

Why am I getting these strange characters instead of simply getting the characters "aaaa"?


Edit:

These are the characters displayed in Konsole:

幡`懾䁞幡`懾䁞

  • You're seeing mojibake. What is your shell's LANG variable set to? As well, could you copy and paste the characters that you saw? With them, it's possible to identify what's going on, but it's difficult (for me, at least) to identify CJK code points from a picture. – ErikF Jun 9 '18 at 5:26
  • @ErikF "What is your shell's LANG variable set to?" I am not using a shell, I am using the xxd program, and it is not xxd that is echoing back what I write in Konsole, it is the line discilpline that is doing that (I know that because the echo termios flag is set). "could you copy and paste the characters that you saw" I edited my question and did that. – rony_t Jun 9 '18 at 6:02
  • A related question is unix.stackexchange.com/questions/446355 . – JdeBP Jun 9 '18 at 7:58
1

Let's even further emphasize that xxd -p is irrelevant, we're not talking about its output. It hasn't even seen the input due to line buffering on the kernel's side, and hence hasn't produced any output. For what it's worth, it could also be a cat or a sleep 100000 or whatever. We're talking about how the kernel (the line discipline) echoes back the input.

If you switch back to UTF-8 and then press an Enter to xxd -p, its output goes like fffe6100fffe6100. So little endian is confirmed (or probably it's the architecture's native byte order), but surprisingly, there's a BOM before every character. Makes me suspect that it wasn't properly thought through by Konsole developers, they just blindly invoke iconv with UTF-16 (without BE or LE specified) as the target character set, for each chunk of input as it becomes available, and iconv places it there.

Let's strace konsole to see what it does on its file descriptor connected to /dev/ptmx:

write(..., "\377\376a\0", 4)             = 4
[...]
read(..., "\377\376a^@", 5)              = 5

The NUL byte (0x00) comes back as literal ^@, that is, 0x5e followed by 0x40.

Along with a (0x61), this gives you U+5e61, which is exactly the first glyph you see appearing. Plus, you'll be off by a byte, that is, the next supposedly low byte will be interpreted as high byte, and vice versa.

For the byte 0x00 the kernel simply mangles how it's echoed back. For some other bytes it performs other actions as well. For example, the byte 0x03 (^C) typically triggers an interrupt to be sent to the foreground process, 0x15 (^U) wipes out the data you've entered so far, 0x0a and/or 0x0d (i.e. newline) flushes the data to your app, etc. All these bytes can (and do) legally occur within the UTF-16 representation of characters, and you sure don't want any of these happening while typing your input.

In order to use UTF-16 on the line disciple, the kernel would need to provide explicit support for this and would need to be told that this encoding is being used (something along the lines of an stty utf16). To my best knowledge, this is not implemented (luckily – it would be a total waste of developer resources). The kernel expects an ASCII-compatible encoding to be used, which UTF-16 isn't.

Even if UTF-16 was implemented in the kernel for the tty line, the entire ecosystem would necessarily be quite fragile. The terminal can receive data from multiple sources simultaneously, and there's no way to guarantee that all producers of data and all transporters (e.g. ssh) can always keep the bytes coupled in pairs. Once it goes off by a byte (as seen above), the rest is unusable.

I'm now even more certain that Konsole developer didn't properly think about it. In my opinion UTF-16 should be removed from their list of offered encodings, or at least a warning should be shown. I've submitted Konsole bug 395171.

  • "but surprisingly, there's a BOM before every character" No, there is no BOM. What is happening is the following: The character is encoded in UTF-16 little endian as 40 FF, and the character is encoded in UTF-16 little endian as FE 61 and so when they are next to each other they become 40 FF FE 61. – rony_t Jun 9 '18 at 11:15
  • There's already a BOM in the data that konsole sends to the kernel, see that \377\376 in the write() call. It comes back from the kernel as-is, which, after the first NUL -> ^@ conversion, is misaligned at an odd byte count offset and gets interpreted (displayed) by konsole as ` and 懾 as you say. In my answer I explained to you in details what goes wrong before this. – egmont Jun 9 '18 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.