I have 500 files that contain escaped Unicode characters like so:
Albert Vel\u00e1zquez V\u00e9ronique Ekin
etc. The data was queried from a server by a script through
XMLHttpRequests and I assume the type of quoting stems from there.
As you can see each
\u00xx needs to be replaced by the respective special character:
\u00e1 -> á \u00e9 -> é
Question: How do I replace these code sequences by their respective UTF-8 counterpart, non-interactively within all files?
The Unicode code points seem to be all 8-bit but it was not possible to check all occurrences (too many). Perhaps multi-byte characters would be displayed like
\u00xx\u00yy? Or perhaps
\uxxyy if this is some sort of 7-bit clean UTF-16? I hope someone here recognises the character representation (I could not find it online) and can recommend a script that handles multi-byte sequences correctly.
Please note that
iconv cannot handle the file format:
# iconv --list | wc -l 1179
iconv knows 1179 encodings. Just try them all out:
# foreach enc ( `iconv --list | tr -d /` ) foreach? echo ==== $enc >> enctest foreach? echo 'Vel\u00e1zquez' | iconv -f $enc -t UTF-8 >> enctest foreach? end # grep -a --before=1 Velázquez enctest Exit 1
The file does not contain the correct string :-(