Convert an ASCII file with octal escapes for UTF-8 codes to UTF-8

Question

1. I have a text file which is an ASCII file itself, but contains octal escape sequences representing codes in utf-8:

   \350\207\252\345\212\250\346\216
   

   Is there some program or command that can convert such ASCII file to a text file actually encoded in utf-8?

2. By the way, this site is "Online ASCII(Unicode Escaped) to Unicode(UTF-8) converter tool", and this site is "Online Unicode(UTF-8) to ASCII(Unicode Escaped) converter tool". Do they make the conversion in my question? If not, what kinds of conversion do they make?

Answer 1

If you have these escape sequences in a shell variable, in dash, mksh or bash:

printf %b "$string_with_backslash_escapes"

This isn't POSIX: the %b specifier is POSIX but it requires a 0 after each backslash. This also interprets other backslash escapes: \n as a newline, \t as a tab, etc.

Here's a perl one-liner that converts octal escape sequences only.

perl -pe 's[\\(?:([0-7]{1,3})|(.))] [defined($1) ? chr(oct($1)) : $2]eg'

http://www.rapidmonkey.com/unicodeconverter/reverse.jsp interprets octal values as Latin-1 characters, I don't know why Unicode and UTF-8 are mentioned in the page. I have no idea what http://www.rapidmonkey.com/unicodeconverter/advanced.jsp does.

Answer 2

Using just Bash:

3.1.2.4 ANSI-C Quoting

Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard. Backslash escape sequences, if present, are decoded as follows:

…

<i>nnn the eight-bit character whose value is the octal value nnn (one to three digits)

Demonstration in a UTF-8 terminal:

$ echo $'\350\207\252\345\212\250\346\216'
自动?

The last character displays as a question mark because the sequence is malformed: only two of the three required bytes are present.

---

The website you linked to performs RFC 5137 encoding/decoding.

If you enter \u81ea\u52a8 in the "ASCII (Unicode Escaped)" text area, you'll get 自动 as output, because 自 is Unicode Character U+81EA (whose UTF-8 representation is e8 87 aa in hex, or 350 207 252 in octal) and 动 is Unicode character U+52A8 (whose UTF-8 representation is e5 8a a8 in hex, or 345 212 250 in octal).

Answer 3

Python in the interactive shell can do at least some of this. But the sequence above appears to be corrupted:

wilmer@ruby:~$ python
Python 2.7.10 (default, Jul  1 2015, 10:54:53) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> unicode("\350\207\252\345\212\250\346\216", "utf-8")
Traceback (most recent call last):
  File "", line 1, in 
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-7: unexpected end of data
>>> unicode("\350\207\252\345\212\250", "utf-8")
u'\u81ea\u52a8'
>>> print unicode("\350\207\252\345\212\250", "utf-8")
自动

Answer 4

Simplest way is ascii2uni -a K, for example:

cat escaped.txt | ascii2uni -a K > unescaped.txt

Answer 5

I want to add that with some software it may be enough to just select, copy, paste the escaped text. For example, Wireshark shows UTF-8 text as a sequence of octal-escaped bytes, but when you copy & paste it into Gedit, you get normal text, e.g. \341\203\231 becomes კ.