2

I have 500 files that contain escaped Unicode characters like so:

Albert Vel\u00e1zquez
V\u00e9ronique Ekin

etc. The data was queried from a server by a script through XMLHttpRequests and I assume the type of quoting stems from there.

As you can see each \u00xx needs to be replaced by the respective special character:

\u00e1 -> á
\u00e9 -> é

etc.

Question: How do I replace these code sequences by their respective UTF-8 counterpart, non-interactively within all files?

The Unicode code points seem to be all 8-bit but it was not possible to check all occurrences (too many). Perhaps multi-byte characters would be displayed like \u00xx\u00yy? Or perhaps \uxxyy if this is some sort of 7-bit clean UTF-16? I hope someone here recognises the character representation (I could not find it online) and can recommend a script that handles multi-byte sequences correctly.

EDIT: Please note that iconv cannot handle the file format:

# iconv --list | wc -l
1179

iconv knows 1179 encodings. Just try them all out:

# foreach enc ( `iconv --list | tr -d /` )
foreach? echo ==== $enc >> enctest
foreach? echo 'Vel\u00e1zquez' | iconv -f $enc -t UTF-8 >> enctest
foreach? end
# grep -a --before=1 Velázquez enctest
Exit 1

The file does not contain the correct string :-(

4
  • 1
    I bet the answer will involve iconv
    – Stewart
    Apr 12, 2021 at 14:55
  • @Stewart Yes, I tried that actually but iconv could not handle the format (though the man page seemed to suggest it).
    – Ned64
    Apr 12, 2021 at 16:03
  • Which shell are you using?. It could be converted back by a few shells.
    – user232326
    Apr 12, 2021 at 17:59
  • @Isaac Apparently the tcsh's builtin echo command cannot do it natively, nor can /bin/echo from current GNU coreutils.
    – Ned64
    Apr 13, 2021 at 7:30

1 Answer 1

2

One option would be to parse it through echo (or printf).

#!/bin/bash
IFS=
while read -r line ; do
  printf '%b\n' "$line"
  #or: echo -e "$line"
  #or: printf "$line\n"
done <infile >outfile

WARNING: This method will interpret ALL escaped charaters! So it will only work if the ONLY occurrences of backslashes are for unicode characters or uniterpretable .

Example

$file infile
input:  ASCII text
$cat infile
Albert Vel\u00e1zquez
V\u00e9ronique Ekin
DOS Path: C:\data\user\file\u123.txt

$file outfile
outfile: UTF-8 Unicode text
$cat outfile
Albert Velázquez
Véronique Ekin
DOS Path: C:\data\user
                      ileǵ.txt

Where \f is interpreted as form feed, \u123 as ǵ, while \d and \u(ser) do not have interpretations.

1
  • Thanks, that works well (my file is not very special, no NULs and no other backslashes as these are files with database entries about people). I have incorporated LL3's suggestions (hopefully understood them correctly, feel free to edit).
    – Ned64
    Apr 12, 2021 at 18:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .