How can I examine the Unicode encoding of a text document

Question

Given a text document or string in an encoding such as UTF-8, what is the easiest way to find out what Unicode encoding represents the characters I see?

I should like to find out which code points have been used, particularly for almost identical characters, such as Á (Latin) and А́ (Cyrillic). I should also like to know how they have been converted to UTF-8. I should prefer a method that works in an editor such as kate, but a command line alternative is acceptable.

What I have tried

I have tried running the text through od -xc, but realised that this was not terribly helpful, as UTF-8 encoding rearranges the bits among bytes. The case study below illustrates for the case of accented Cyrillic vowels how laborious this can be.

I use a text document including many useful Unicode characters for cutting and pasting, in a format I find more convenient than the usual Character Map programmes supply, and I usually have this open in kate. Since I should like to inspect the encodings in such a file, an option in an editor would be useful.

What I hope for

When I was using the programmable editor TPU on VMS, it was straightforward to programme it to add a function to display under the line of text another two lines showing the vertically arranged hexadecimal codes of the bytes of the file (with proper spacing for tabs). This would be a useful possibility, though for UTF-8 one would need up to 8 lines.

Another possibility would be to select a character and have a function to pop up a window showing the code points used to produce it and/or how these are converted into bytes by UTF-8. In this case one could show more information, but it would be less useful for visually scanning details of a longer text.

Instead of a function of an editor, one could have a new format for od or for a derivative thereof.

Case study: Accented Russian vowels

I am currently learning Russian, and want to type Cyrillic characters with accents. I have collected the Russian vowels with accents from sources such as Wiktionary (e.g. попугай), giving me this text:

Á á Ó ó É é У́ ý И́ и́ Ы́ ы́ Э́ э́ Ю́ ю́ Я́ я́

I gather from this answer to my question below¹ that these accented vowels have to be represented using U+0301 COMBINING ACUTE ACCENT. When I run this through od -tx1 -tc -w26 (-tc so spaces show up, width 26 to fit this site), what I get is:

0000000  c3  81  20  c3  a1  20  c3  93  20  c3  b3  20  c3  89  20  c3  a9  20  d0  a3  cc  81  20  c3  bd  20
        303 201     303 241     303 223     303 263     303 211     303 251     320 243 314 201     303 275    
0000032  d0  98  cc  81  20  d0  b8  cc  81  20  d0  ab  cc  81  20  d1  8b  cc  81  20  d0  ad  cc  81  20  d1
        320 230 314 201     320 270 314 201     320 253 314 201     321 213 314 201     320 255 314 201     321
0000064  8d  cc  81  20  d0  ae  cc  81  20  d1  8e  cc  81  20  d0  af  cc  81  20  d1  8f  cc  81  0a
        215 314 201     320 256 314 201     321 216 314 201     320 257 314 201     321 217 314 201

Kate tells me that my file is in UTF-8 (under Tools Encoding Unicode). UTF-8 represents Cyrillic characters and combining accents, since they lie in the range U+0080..U+07FF, as 2 bytes; for such a code point the lowest 11 bits are represented in those two bytes as 110xxxxx₂ 10xxxxxx₂, where 1 & 0 are prescribed, and x is a bit from the code point. In other words,

• the first byte = 0xc0 + x,
• the second byte = 0x80 + y, where
• x and y are the quotient and remainder when the code point is divided by 2⁶ = 0x40, i.e. the upper 10 bits (of which but 5 are non-zero) and lower 6 bits.

Since the code point for the combining acute accent is U+0301, its bits are 0011 00 00 0001 meaning x=0x0c and y=0x01, so it is represented by the bytes 0xcc 0x81. Similarly,

Cyrillic “Á” → U+0410 → 0100 00 01 0000 → 0x10 0x10 → 0xd0 0x90.
Cyrillic “а́” → U+0430 → 0100 00 11 0000 → 0x10 0x20 → 0xd0 0xb0.

which disagrees with the dump output, but

Latin “Á” → U+00C1 → 0000 11 00 0001 → 0x03 0x01 → 0xc3 0x81.

So my saved Á á Ó ó É é were in fact Latin, as all six only use two bytes, so no combining accent!

---

Notes

¹ _{I have asked about this in How can I type Russian characters with accents in KDE?. An answer to made me realise that I wanted the functionality described in this question.}

Answer 1

Emacs

C-x = (M-x what-cursor-position) shows code point information about the current character at the bottom of the screen, for example

Char: И (1048, #o2030, #x418, file ...) point=7 of 8 (75%) column=0

C-u C-x = opens a window with more information, including the code point, the byte representation, metadata about the Unicode character, what font is used to display the character, etc.

             position: 7 of 8 (75%), column: 0
            character: И (displayed as И) (codepoint 1048, #o2030, #x418)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x0418
               script: cyrillic
               syntax: w    which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #xD0 #x98
            file code: #xD0 #x98 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-DAMA-Ubuntu Mono-normal-normal-normal-*-17-*-*-*-m-0-iso10646-1 (#x2CB)
         Unicode data:
                 Name: CYRILLIC CAPITAL LETTER I
             Category: Letter, Uppercase
      Combining class: Lu
        Bidi category: Lu
             Old name: CYRILLIC CAPITAL LETTER II
            Lowercase: и

Character code properties: customize what to show
  name: CYRILLIC CAPITAL LETTER I
  old-name: CYRILLIC CAPITAL LETTER II
  general-category: Lu (Letter, Uppercase)
  decomposition: (1048) ('И')

[back]

Command line

The unicode utility (available in some distributions such as Debian/Ubuntu/…, or with pip3 install unicode) shows information about one or more Unicode characters. Note that if you copy-paste from an editor, that editor may encode the clipboard differently from the file.

$ unicode И
U+0418 CYRILLIC CAPITAL LETTER I
UTF-8: d0 98 UTF-16BE: 0418 Decimal: И Octal: \02030
И (и)
Lowercase: 0438
Category: Lu (Letter, Uppercase)
Bidi: L (Left-to-Right)