1

I have 500 files that contain escaped Unicode characters like so:

Albert Vel\u00e1zquez
V\u00e9ronique Ekin

etc. The data was queried from a server by a script through XMLHttpRequests and I assume the type of quoting stems from there.

As you can see each \u00xx needs to be replaced by the respective special character:

\u00e1 -> á
\u00e9 -> é

etc.

Question: How do I replace these code sequences by their respective UTF-8 counterpart, non-interactively within all files?

The Unicode code points seem to be all 8-bit but it was not possible to check all occurrences (too many). Perhaps multi-byte characters would be displayed like \u00xx\u00yy? Or perhaps \uxxyy if this is some sort of 7-bit clean UTF-16? I hope someone here recognises the character representation (I could not find it online) and can recommend a script that handles multi-byte sequences correctly.

EDIT: Please note that iconv cannot handle the file format:

# iconv --list | wc -l
1179

iconv knows 1179 encodings. Just try them all out:

# foreach enc ( `iconv --list | tr -d /` )
foreach? echo ==== $enc >> enctest
foreach? echo 'Vel\u00e1zquez' | iconv -f $enc -t UTF-8 >> enctest
foreach? end
# grep -a --before=1 Velázquez enctest
Exit 1

The file does not contain the correct string :-(

4
  • 1
    I bet the answer will involve iconv – Stewart Apr 12 at 14:55
  • @Stewart Yes, I tried that actually but iconv could not handle the format (though the man page seemed to suggest it). – Ned64 Apr 12 at 16:03
  • Which shell are you using?. It could be converted back by a few shells. – ImHere Apr 12 at 17:59
  • @Isaac Apparently the tcsh's builtin echo command cannot do it natively, nor can /bin/echo from current GNU coreutils. – Ned64 Apr 13 at 7:30
3

One option would be to parse it through echo (or printf).

#!/bin/bash
IFS=
while read -r line ; do
  printf '%b\n' "$line"
  #or: echo -e "$line"
  #or: printf "$line\n"
done <infile >outfile

WARNING: This method will interpret ALL escaped charaters! So it will only work if the ONLY occurrences of backslashes are for unicode characters or uniterpretable .

Example

$file infile
input:  ASCII text
$cat infile
Albert Vel\u00e1zquez
V\u00e9ronique Ekin
DOS Path: C:\data\user\file\u123.txt

$file outfile
outfile: UTF-8 Unicode text
$cat outfile
Albert Velázquez
Véronique Ekin
DOS Path: C:\data\user
                      ileǵ.txt

Where \f is interpreted as form feed, \u123 as ǵ, while \d and \u(ser) do not have interpretations.

1
  • Thanks, that works well (my file is not very special, no NULs and no other backslashes as these are files with database entries about people). I have incorporated LL3's suggestions (hopefully understood them correctly, feel free to edit). – Ned64 Apr 12 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.