11

I am trying to identify a strange character I have found in a file I am working with:

$ cat file
�
$ od file
0000000 005353
0000002
$ od -c file
0000000 353  \n
0000002
$ od -x file
0000000 0aeb
0000002

The file is using ISO-8859 encoding and can't be converted to UTF-8:

$ iconv -f ISO-8859 -t UTF-8 file
iconv: conversion from `ISO-8859' is not supported
Try `iconv --help' or `iconv --usage' for more information.
$ iconv  -t UTF-8 file
iconv: illegal input sequence at position 0
$ file file
file: ISO-8859 text

My main question is how can I interpret the output of od here? I am trying to use this page which lets me translate between different character representations, but it tells me that 005353 as a "Hex code point" is which doesn't seem right and 0aeb as a "Hex code point" is which, again, seems wrong.

So, how can I use any of the three options (355, 005353 or 0aeb) to find out what character they are supposed to represent?

And yes, I did try with Unicode tools but it doesn't seem to be a valid UTF character either:

$ uniprops $(cat file)
U+FFFD ‹�› \N{REPLACEMENT CHARACTER}
    \pS \p{So}
    All Any Assigned Common Zyyy So S Gr_Base Grapheme_Base Graph X_POSIX_Graph
       GrBase Other_Symbol Print X_POSIX_Print Symbol Specials Unicode

if I understand the description of the Unicode U+FFFD character, it isn't a real character at all but a placeholder for a corrupted character. Which makes sense since the file isn't actually UTF-8 encoded.

  • 5
    EB could be δ in code page 437, or Ù in code page 850, or ë in 8859-1; would any of those make sense? (iconv complains because you didn’t specify the source character set, so it uses your default which is probably UTF-8.) – Stephen Kitt Apr 28 '17 at 13:06
  • @StephenKitt yes, ë is what I see when the data is used on another program! But how can I know this? Isn't it somewhere in the data I provide? How did you find it? Oh I had tried iconv with -f ISO-8859 but it complained about conversion from ISO-8859' is not supported`. – terdon Apr 28 '17 at 13:09
  • 1
    Argh! I see, I needed to use just eb and ignore the 0x hex indicator or whatever that is. My ignorance of this sort of thing is deep. Could you post an answer explaining that @StephenKitt? – terdon Apr 28 '17 at 13:10
  • 5
    Your crucial mistake here is that ISO-8859 is not the name of an encoding. It's a family of encodings; apparently, the one you are looking for is ISO-8859-1. – tripleee Apr 28 '17 at 15:30
  • 1
    Then your iconv would have succeeded; and/or you could have looked it up e.g. on Wikipedia. For this very specific encoding, fileformat.info/info/unicode/char/00eb/index.htm also works (Unicode is equivalent to ISO-8859-1 in the range 128-255, though of course no UTF encoding is compatible with it). – tripleee Apr 28 '17 at 16:22
22

Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë:

$ printf "\353\n" | iconv -f ISO-8859-1
ë

Other candidates would be δ in code page 437, Ù in code page 850...

od -x’s output is confusing in this case because of endianness; a better option is -t x1 which uses single bytes:

$ printf "\353\n" | od -t x1
0000000 eb 0a
0000002

od -x maps to od -t x2 which reads two bytes at a time, and on little-endian systems outputs the bytes in reverse order.

When you come across a file like this, which isn’t valid UTF-8 (or makes no sense when interpreted as a UTF-8 file), there’s no fool-proof way to automatically determine its encoding (and character set). Context can help: if it’s a file produced on a Western PC in the last couple of decades, there’s a fair chance it’s encoded in ISO-8859-1, -15 (the Euro variant), or Windows-1252; if it’s older than that, CP-437 and CP-850 are likely candidates. Files from Eastern European systems, or Russian systems, or Asian systems, would use different character sets that I don’t know much about. Then there’s EBCDIC... iconv -l will list all the character sets that iconv knows about, and you can proceed by trial and error from there.

(At one point I knew most of CP-437 and ATASCII off by heart, them were the days.)

  • 1
    OK, in the wikipedia page you link to, I can see that ë is described as 00EB and 234. What are those extra 00? And why isn't it 355 as I expected from the od output? I am trying to get a more general answer about how I can use the od output to identify the character. Could you maybe explain something about interpreting hex codes and/or what information is needed to be able to identify an unknown character (encoding and whatever else)? – terdon Apr 28 '17 at 13:18
  • EB is 353 in octal (not 355). I’ll try to generalise... – Stephen Kitt Apr 28 '17 at 13:19
  • Whoops, sorry, I meant 353. So the 353 is an octal representation, not decimal. Argh. – terdon Apr 28 '17 at 13:19
  • 1
    Yes, the “o” in od stands for octal ;-). – Stephen Kitt Apr 28 '17 at 13:20
  • 1
    In any case, the (U+FFFD) would be displayed by the terminal emulator as a substitute for that 0xeb byte that doesn't form a valid character in UTF-8. It's not clear why uniprops $(cat file) (missing quotes btw) would report that (I don't know about that uniprops command). unicode "$(cat file)" on Debian does output Sequence '\xeb' is not valid in charset 'UTF-8' as I'd expect. – Stéphane Chazelas Apr 28 '17 at 13:27
5

Note that od is short for octal dump, so 005353 are the two bytes as octal word, od -x is 0aeb in hexadecimal as word, and the actual contents of your file are the two bytes eb and 0a in hexadecimal, in this order.

So both 005353 and 0aeb can't just be interpreted as "hex code point".

0a is a line feed (LF), and eb depends on your encoding. file is just guessing the encoding, it could be anything. Without any further information where the file came from etc. it will be difficult to find out.

  • I realize this is because I don't understand how code points (or hex, really) work, but how can I know this? I usually use od -c since that produces output I can understand. How could I have used the 355 that produces to identify the character? And why is it printing 0aeb instead of eb0a if 0a is the newline? – terdon Apr 28 '17 at 13:13
  • @terdon endianness... See my updated answer. – Stephen Kitt Apr 28 '17 at 13:18
2

It is impossible to guess with 100% of accuracy the charset of text files.

Tools like chardet, firefox, file -i when there is no explicit charset information defined (eg. if a HTML contains a meta charset=... in the head, things are easier) will try to use heuristics that are not so bad if the text is big enough.

In the following, I demonstrate charset-detection with chardet (pip install chardet / apt-get install python-chardet if necessary).

$ echo "in Noël" | iconv -f utf8 -t latin1  | chardet
<stdin>: windows-1252 with confidence 0.73

After having good charset candidate, we can use iconv, recode or similar to change the file charset to your "active" charset (in my case utf-8) and see if it guessed correctly...

iconv -f windows-1252  -t utf-8 file

Some charset (like iso-8859-3, iso-8859-1) have many chars in common -- sometimes it is not easy to see if we found the perfect charset...

So it is very important to have metadata associated with relevant text (eg XML).

  • Hmm. I can't reproduce it here, it just crashes. But in any case, isn't that simply telling me the encoding of the file? My issue is identifying the character not the file's encoding. That I already knew. – terdon Apr 28 '17 at 15:17
  • 1
    Sorry, I failed to understood the question (my usual problem is identifying the charset) . if you now the encoding, iconv -f ... -t utf-8 will show you the chars? – JJoao Apr 28 '17 at 15:41
  • No. I show the encoding right there. There was one particular character not supported by that encoding and it is that character I was trying to identify. – terdon Apr 28 '17 at 15:42
  • 1
    Iso-8859 is not the encoding! the encoding is iso-8850-1 . iso-8859 is a iso standart the includes several chaset definitions. Try file -i ... – JJoao Apr 28 '17 at 15:44
  • 1
    @terdon, sorry to insist, but, all the tricks you tried work with the right charset. Ex: iconv -f ISO-8859-1 -t UTF-8 file – JJoao Apr 28 '17 at 15:53
0
#!/bin/bash
#
# Search in a file, a known (part of a ) String (i.E.: Begrüßung),
# by testing all encodings
#
[[ $# -ne 2 ]] && echo "Usage: encoding-finder.sh FILE fUnKy_CHAR_FOR_SURE_IN_FILE" && exit
FILE=$1
PATTERN=$2
for enc in $( iconv -l | sed 's/..$//') 
do 
    iconv -f $enc -t UTF-8 $FILE  2>/dev/null | grep -m 1 $PATTERN && echo $enc 
done 

If I get a file, which contains, for Instance the Word Begrung, I can infer that Begrüßung might be meant. So I convert it by all known encodindgs and look, whether one is found, which converts it properly.

Usually, there are multiple encodings which seem to fit.

For longer files, you might cut a snippet instead of converting hundreds of pages.

So I would call it

encodingfinder.sh FILE Begrüßung

and the script tests, whether by converting it with the known encodings, which of them produce "Begrüßung".

To find such characters, less is usually of help, since funky characters often stand out. From the context, the right word to search for can usually be inferred. But we don't want to check with a hexeditor, what byte this is, and then visit endless tables of encodings, to find our offender. :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.