I have 500 files that contain escaped Unicode characters like so:
Albert Vel\u00e1zquez
V\u00e9ronique Ekin
etc. The data was queried from a server by a script through XMLHttpRequest
s and I assume the type of quoting stems from there.
As you can see each \u00xx
needs to be replaced by the respective special character:
\u00e1 -> á
\u00e9 -> é
etc.
Question: How do I replace these code sequences by their respective UTF-8 counterpart, non-interactively within all files?
The Unicode code points seem to be all 8-bit but it was not possible to check all occurrences (too many). Perhaps multi-byte characters would be displayed like \u00xx\u00yy
? Or perhaps \uxxyy
if this is some sort of 7-bit clean UTF-16? I hope someone here recognises the character representation (I could not find it online) and can recommend a script that handles multi-byte sequences correctly.
EDIT:
Please note that iconv
cannot handle the file format:
# iconv --list | wc -l
1179
iconv
knows 1179 encodings. Just try them all out:
# foreach enc ( `iconv --list | tr -d /` )
foreach? echo ==== $enc >> enctest
foreach? echo 'Vel\u00e1zquez' | iconv -f $enc -t UTF-8 >> enctest
foreach? end
# grep -a --before=1 Velázquez enctest
Exit 1
The file does not contain the correct string :-(
iconv
iconv
could not handle the format (though theman
page seemed to suggest it).tcsh
's builtinecho
command cannot do it natively, nor can/bin/echo
from current GNU coreutils.