Chinese characters instead of Latin being written to file

Question

When I run sed like this and print to a console everything is fine:

sed '/Q/{
s/Q//g
r /Users/ericbrotto/Desktop/question.txt
}' Commision.txt

But when I do this and output t a file:

sed '/Q/{
s/Q//g
r /Users/ericbrotto/Desktop/question.txt
}' Commision.txt > newFile

...my new string (the one that was properly replaced in the previous output) now reads as a bunch of asian (I believe Mandarin) characters.

Any ideas?

_{This is a follow up problem to a question I posed earlier.}

Answer 1

I have previously noticed that if you take ASCII encoded text (or, equivalently, UTF-8 encoded ASCII text) and decode it as UTF-16, you often get “Chinese characters” (different ones depending on whether you decode it as UTF-16BE or UTF-16LE). Based on this, I think you are dealing with mixed encodings. My guess is that Commision.txt is encoded as either UTF-16BE or UTF-16LE, that question.txt is plain ASCII (or UTF-8 encoded ASCII), and that your newFile ends up as an invalid combination of the encodings from both files.

Things should work better if you use the same encoding in both files; probably UTF-8 will work best. If you need the final output to be in some other encoding, then you can use iconv to convert it (iconv -f UTF-8 -t UTF-16BE <newFile >newfile.utf16be.txt).

---

Effectively, the UTF-16 encoding of ASCII characters is the same as the ASCII encoding but with extra NUL characters inserted between each ASCII character along with one more NUL before or after the whole lot (depending on the endianness of the UTF-16 encoding). This means that ASCII text encoded as either UTF-8, or UTF-16 will look “normal” when directly viewed on a UTF-8 terminal (i.e. your “print to a console”).

As long as the file contents are kept separate, any encoding-detecting viewing environment (e.g. an editor) will probably correctly detect the encoding (or at least pick one that is close enough considering that UTF-8 and many single-byte encodings are identical in the ASCII range).

But, you have sed mixing the files together. Unfortunately, sed is not “smart” enough to realize that it is dealing with files using two different text encodings. You end up with (according to my guess) a file that is mostly UTF-16 encoded (from Commision.txt) with a UTF-8 encoded section (from question.txt) in the middle (or wherever your Q puts it). The result is probably invalid if wholly decoded as UTF-8, but possibly valid when wholly decoded as UTF-16 (though with some unexpected contents where the UTF-8 data is).

---

Here is an example:

Commision.txt is UTF-16BE encoded ASCII (with a BOM).

% xxd Commision.txt 
0000000: feff 0046 0069 0072 0073 0074 0020 006c  ...F.i.r.s.t. .l
0000010: 0069 006e 0065 000a 004c 0069 006e 0065  .i.n.e...L.i.n.e
0000020: 0020 0077 0069 0074 0068 0020 0061 0020  . .w.i.t.h. .a. 
0000030: 0075 0063 0020 0027 0071 0027 003a 0020  .u.c. .'.q.'.:. 
0000040: 0028 0051 0029 000a 004c 0061 0073 0074  .(.Q.)...L.a.s.t
0000050: 0020 006c 0069 006e 0065 000a            . .l.i.n.e..

question.txt is ASCII (or UTF-8 encoded ASCII).

% xxd question.txt
0000000: 5768 6174 2069 7320 7468 6520 6169 722d  What is the air-
0000010: 7370 6565 6420 7665 6c6f 6369 7479 206f  speed velocity o
0000020: 6620 616e 2075 6e6c 6164 656e 2073 7761  f an unladen swa
0000030: 6c6c 6f77 3f0a                           llow?.

I combine them with sed.

% sed '/Q/{
s/Q//g
r question.txt
}' Commision.txt >newFile

newFile is a mess.

sed deleted the Q as a single byte (51) instead of its two byte UTF-16 representation (00 51).
This ruins the two-byte alignment of the rest of the file, gives a whole length that is odd instead of even, and introduces a UTF-16 NULL (0000).

% xxd newFile
0000000: feff 0046 0069 0072 0073 0074 0020 006c  ...F.i.r.s.t. .l
0000010: 0069 006e 0065 000a 004c 0069 006e 0065  .i.n.e...L.i.n.e
0000020: 0020 0077 0069 0074 0068 0020 0061 0020  . .w.i.t.h. .a. 
0000030: 0075 0063 0020 0027 0071 0027 003a 0020  .u.c. .'.q.'.:. 
0000040: 0028 0000 2900 0a57 6861 7420 6973 2074  .(..)..What is t
0000050: 6865 2061 6972 2d73 7065 6564 2076 656c  he air-speed vel
0000060: 6f63 6974 7920 6f66 2061 6e20 756e 6c61  ocity of an unla
0000070: 6465 6e20 7377 616c 6c6f 773f 0a00 4c00  den swallow?..L.
0000080: 6100 7300 7400 2000 6c00 6900 6e00 6500  a.s.t. .l.i.n.e.
0000090: 0a                                       .

Despite the mess, it looks fine in my UTF-8 terminal.

% cat newFile
First line
Line with a uc 'q': ()
What is the air-speed velocity of an unladen swallow?
Last line

When I load it in Vim, however things are obviously amiss (there is actually a NUL after the open parenthesis, but its presence caused this post be truncated). Vim warns “CONVERSION ERROR in line 2”.

First line
Line with a uc 'q': (⤀੗桡琠楳⁴桥⁡楲⵳灥敤⁶敬潣楴礠潦⁡渠畮污摥渠獷慬汯眿਀䰀愀猀琀 氀椀渀攀

If I delete the question mark from question.txt (to give an even number of bytes again) and regenerate newFile, then I get the last line “back” (though it is stuck to the end of the second line) and avoid the conversion warning from Vim.

First line
Line with a uc 'q': (⤀੗桡琠楳⁴桥⁡楲⵳灥敤⁶敬潣楴礠潦⁡渠畮污摥渠獷慬汯眊Last line