Given a text document or string in an encoding such as UTF-8, what is the easiest way to find out what Unicode encoding represents the characters I see?
I should like to find out which code points have been used, particularly for almost identical characters, such as Á (Latin) and А́ (Cyrillic). I should also like to know how they have been converted to UTF-8. I should prefer a method that works in an editor such as
kate, but a command line alternative is acceptable.
What I have tried
I have tried running the text through
od -xc, but realised that this was not terribly helpful, as UTF-8 encoding rearranges the bits among bytes.
The case study below illustrates for the case of accented Cyrillic vowels how laborious this can be.
I use a text document including many useful Unicode characters for cutting and pasting, in a format I find more convenient than the usual Character Map programmes supply, and I usually have this open in
Since I should like to inspect the encodings in such a file,
an option in an editor would be useful.
What I hope for
When I was using the programmable editor TPU on VMS, it was straightforward to programme it to add a function to display under the line of text another two lines showing the vertically arranged hexadecimal codes of the bytes of the file (with proper spacing for tabs). This would be a useful possibility, though for UTF-8 one would need up to 8 lines.
Another possibility would be to select a character and have a function to pop up a window showing the code points used to produce it and/or how these are converted into bytes by UTF-8. In this case one could show more information, but it would be less useful for visually scanning details of a longer text.
Instead of a function of an editor, one could have a new format for
od or for a derivative thereof.
Case study: Accented Russian vowels
I am currently learning Russian, and want to type Cyrillic characters with accents. I have collected the Russian vowels with accents from sources such as Wiktionary (e.g. попугай), giving me this text:
Á á Ó ó É é У́ ý И́ и́ Ы́ ы́ Э́ э́ Ю́ ю́ Я́ я́
I gather from this answer to my question below1 that these accented vowels have to be represented using
U+0301 COMBINING ACUTE ACCENT. When I run this through
od -tx1 -tc -w26 (
-tc so spaces show up, width 26 to fit this site), what I get is:
0000000 c3 81 20 c3 a1 20 c3 93 20 c3 b3 20 c3 89 20 c3 a9 20 d0 a3 cc 81 20 c3 bd 20 303 201 303 241 303 223 303 263 303 211 303 251 320 243 314 201 303 275 0000032 d0 98 cc 81 20 d0 b8 cc 81 20 d0 ab cc 81 20 d1 8b cc 81 20 d0 ad cc 81 20 d1 320 230 314 201 320 270 314 201 320 253 314 201 321 213 314 201 320 255 314 201 321 0000064 8d cc 81 20 d0 ae cc 81 20 d1 8e cc 81 20 d0 af cc 81 20 d1 8f cc 81 0a 215 314 201 320 256 314 201 321 216 314 201 320 257 314 201 321 217 314 201
Kate tells me that my file is in UTF-8 (under Tools Encoding Unicode). UTF-8 represents Cyrillic characters and combining accents,
since they lie in the range U+0080..U+07FF,
as 2 bytes; for such a code point the lowest 11 bits are represented in those two bytes as
0 are prescribed, and
x is a bit from the code point. In other words,
- the first byte =
- the second byte =
0x80+ y, where
- x and y are the quotient and remainder when the code point is divided by 26 =
0x40, i.e. the upper 10 bits (of which but 5 are non-zero) and lower 6 bits.
Since the code point for the combining acute accent is U+0301, its bits are
00 0001 meaning x=
0x0c and y=
0x01, so it is represented by the bytes
Cyrillic “Á” → U+0410 →
Cyrillic “а́” → U+0430 →
which disagrees with the dump output, but
Latin “Á” → U+00C1 →
So my saved
Á á Ó ó É é were in fact Latin, as all six only use two bytes, so no combining accent!
1 I have asked about this in How can I type Russian characters with accents in KDE?. An answer to made me realise that I wanted the functionality described in this question.