interpret \UXXXXXXXX Unicode codes in text file

Question

I have the following file :

$ cat test
Villes visit\U000000e9es

How can I interpret those \UXXXXXXXX codes, e.g. how can I get :

$ cat test | pipe into something
Villes visitées

Answer 1

With perl:

$ perl -C -pe 's/\\U([[:xdigit:]]{8})/chr hex$1/ge' <yourfile
Villes visitées

Assuming the locale uses UTF-8 as its charmap¹, that would convert \UXXXXXXXX to the UTF-8 encoding of the U+XXXXXXXX character. To get UTF-8 Output regardless of the user's locale, change the -C to -CO.

To convert it to the é character in the correct encoding for the user's locale (assuming there's such a character in the user's locale charset):

perl -Mopen=locale -pe 's/\\U([[:xdigit:]]{8})/chr hex$1/ge' <yourfile

So for instance, in a fr_CH.iso88591 locale, that would convert it to the 0xe9 byte (the encoding of é in ISO8859-1), while in a zh_HK.big5hkscs locale that would convert it to 0x88 0x6d (its encoding in BIG5-HKSCS). And 0xc3 0xa9 in a fr_FR.UTF-8 locale (its UTF-8 encoding). In a ar_AE.iso88596 locale, as ISO8859-6 doesn't have a é character, you'd get Villes visit\x{00e9}es.

Or you could use ICU's uconv (in the icu-devtools package on Debian-based systems) to apply the Hex/C-Any transform :

uconv -x hex/c-any <your-file

It understands \uXXXX and \UXXXXXXXX sequences (more if you use hex-any) and outputs in UTF-8. Pipe to iconv -f utf-8 to get the output in the user's locale (see also iconv's -c option to skip characters that can't be encoded).

$ printf '%s\n' 'é é \x{e9} U+00E9 \u00e9 \U000000e9 \U0001F427 \ud83d\udc27' | uconv -x hex/c-any
é é \x{e9} U+00E9 é é 🐧 🐧
$ printf '%s\n' 'é é \x{e9} U+00E9 \u00e9 \U000000e9 \U0001F427 \ud83d\udc27' | uconv -x hex-any  
é é é é é é 🐧 🐧

(both also recognise java-style surrogate pairs though that shouldn't occur in your output if it's from cpp -P).

For the perl one to understand both \uXXXX and \UXXXXXXXX like uconv's hex/c-any does, change the perl code to:

s/(?|\\u([[:xdigit:]]{4})|\\U([[:xdigit:]]{8}))/chr hex$1/ge

zsh's print builtin also understands those \uXXXX and \UXXXXXXXX (does not require all the 4/8 digits) and many more, so you could also do:

print -- "$(<your-file)"

You'll get an error if there are characters not present in the locale's charmap.

Some printf implementations also support them for their %b format directive:

printf '%b\n' "$(cat <your-file)"

Like zsh's print, it supports more than just the \u/\U ones, at least the \n/\b/\r..., the \0ooo ones and often more like \xHH.

---

^{¹ See output of the locale charmap command; in locales that use other charmaps, what you get is mostly useless in your case. If all the code points on a line are in the 0x0 .. 0xff range, you get the ISO8859-1 encoding (the byte value of the code point), and if not (if there's at least one code point above 0xff in the line), the UTF-8 encoding (and some warnings about it)}