2

I'm running iwlist wlo1 scan | grep ESSID inside a script.

It displays French characters in the following format

\xC3\x89 for É and \xC3\xA9 for é.

I'm not sure what this format is called. I tried using an answer for converting unicode echo -ne '\xC3\xA9' | iconv -f utf-16be but it converted to .

What is the official name for this format and how can I convert it in bash?

  • echo -ne "\xC3\x89" gives É – RomanPerekhrest Oct 28 '17 at 14:10
  • @RomanPerekhrest thank you, that fixes my issue. – Philip Kirkbride Oct 28 '17 at 14:12
  • @RomanPerekhrest do you want to post as an answer? – Philip Kirkbride Oct 28 '17 at 14:14
  • Philip Kirkbride, ok, posted ... – RomanPerekhrest Oct 28 '17 at 14:39
5

Hexdecimal numeric constants are usually represented with 0x prefix.
Character and string constants may express character codes in hexadecimal with the prefix \x followed by two hex digits.

echo -ne '\xC3\x89' should give you É.

  • -e - enable interpretation of backslash escapes(including \xHH - byte with hexadecimal value HH (1 to 2 digits))

To deal with better portability use printf function:

printf "%b" '\xC3\x89'
É
  • echo -e is nonportable -- even bash doesn't always offer it, since when both posix and xpg_echo options are enabled echo -e just prints -e on output, as standard compliance requires. Better to use printf '%b\n' '\xC3\x89' – Charles Duffy Oct 28 '17 at 17:29
  • ...to demonstrate this, consider bash -c 'set -o posix; shopt -s xpg_echo; printf "%b\n" "$1"; echo -e "$1"' _ '\xC3\x89' (and note that xpg_echo can be set on-by-default at compile time). – Charles Duffy Oct 28 '17 at 17:38
  • @CharlesDuffy, you have my update – RomanPerekhrest Oct 28 '17 at 20:05
3

Try

 % echo -n éÉ | hd
 00000000  c3 a9 c3 89                                       |....|
 00000004

where hd is the hexdump utility.

So your thing is just UTF-8 encoding (and your observed output \xC3\x89 is the hexadecimal representation of UTF-8). Today we have UTF-8 everywhere.

My environment has both LANG=en_US.UTF-8 and LC_ALL=en_US.UTF-8 (even if I am French).

Also,

echo '\xC3\x89\xC3\xA9'

produces

Éé

as expected.

And so does

echo  "\\xC3\\x89\\xC3\\xA9" | iconv -t utf-8

Read also locale(7).

Notice that some character encodings (e.g. old plain ASCII) don't have any é or É characters, and others (e.g. ISO-8859-1) encode them differently.

  • When I run echo "\\xC3\\x89\\xC3\\xA9" | iconv -t utf-8 my output is the same as the input, same for normal echo. LANG=en_CA.UTF-8 and LC_ALL not set. I'm hoping to have something that will work on a variety of machines. – Philip Kirkbride Oct 28 '17 at 14:10
  • 1
    You can't display é on an ASCII system, because that letter does not exist in ASCII encoding. – Basile Starynkevitch Oct 28 '17 at 14:12
1

iwlist displays all the bytes that don't correspond to ASCII non-control characters as the \xHH where HH is the byte number in hexadecimal.

To undo that encoding, you can pipe the output of iwlist to

perl -pe 's/\\x([\da-f]{2})/chr(hex($1))/gie'

Byte 0xc3 followed by 0xa9 make up the UTF-8 encoding of the é character.

If your locale also uses the UTF-8 encoding, then the output of perl will show that é character. If not (unlikely as UTF-8 is the norm), you can always pipe the output of perl to iconv -f utf-8 to convert it from UTF-8 to the character encoding used in your locale.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.