2

Given a text document or string in an encoding such as UTF-8, what is the easiest way to find out what Unicode encoding represents the characters I see?

I should like to find out which code points have been used, particularly for almost identical characters, such as Á (Latin) and А́ (Cyrillic). I should also like to know how they have been converted to UTF-8. I should prefer a method that works in an editor such as kate, but a command line alternative is acceptable.

What I have tried

I have tried running the text through od -xc, but realised that this was not terribly helpful, as UTF-8 encoding rearranges the bits among bytes. The case study below illustrates for the case of accented Cyrillic vowels how laborious this can be.

I use a text document including many useful Unicode characters for cutting and pasting, in a format I find more convenient than the usual Character Map programmes supply, and I usually have this open in kate. Since I should like to inspect the encodings in such a file, an option in an editor would be useful.

What I hope for

When I was using the programmable editor TPU on VMS, it was straightforward to programme it to add a function to display under the line of text another two lines showing the vertically arranged hexadecimal codes of the bytes of the file (with proper spacing for tabs). This would be a useful possibility, though for UTF-8 one would need up to 8 lines.

Another possibility would be to select a character and have a function to pop up a window showing the code points used to produce it and/or how these are converted into bytes by UTF-8. In this case one could show more information, but it would be less useful for visually scanning details of a longer text.

Instead of a function of an editor, one could have a new format for od or for a derivative thereof.

Case study: Accented Russian vowels

I am currently learning Russian, and want to type Cyrillic characters with accents. I have collected the Russian vowels with accents from sources such as Wiktionary (e.g. попугай), giving me this text:

Á á Ó ó É é У́ ý И́ и́ Ы́ ы́ Э́ э́ Ю́ ю́ Я́ я́

I gather from this answer to my question below1 that these accented vowels have to be represented using U+0301 COMBINING ACUTE ACCENT. When I run this through od -tx1 -tc -w26 (-tc so spaces show up, width 26 to fit this site), what I get is:

0000000  c3  81  20  c3  a1  20  c3  93  20  c3  b3  20  c3  89  20  c3  a9  20  d0  a3  cc  81  20  c3  bd  20
        303 201     303 241     303 223     303 263     303 211     303 251     320 243 314 201     303 275    
0000032  d0  98  cc  81  20  d0  b8  cc  81  20  d0  ab  cc  81  20  d1  8b  cc  81  20  d0  ad  cc  81  20  d1
        320 230 314 201     320 270 314 201     320 253 314 201     321 213 314 201     320 255 314 201     321
0000064  8d  cc  81  20  d0  ae  cc  81  20  d1  8e  cc  81  20  d0  af  cc  81  20  d1  8f  cc  81  0a
        215 314 201     320 256 314 201     321 216 314 201     320 257 314 201     321 217 314 201

Kate tells me that my file is in UTF-8 (under Tools Encoding Unicode). UTF-8 represents Cyrillic characters and combining accents, since they lie in the range U+0080..U+07FF, as 2 bytes; for such a code point the lowest 11 bits are represented in those two bytes as 110xxxxx2 10xxxxxx2, where 1 & 0 are prescribed, and x is a bit from the code point. In other words,

  • the first byte = 0xc0 + x,
  • the second byte = 0x80 + y, where
  • x and y are the quotient and remainder when the code point is divided by 26 = 0x40, i.e. the upper 10 bits (of which but 5 are non-zero) and lower 6 bits.

Since the code point for the combining acute accent is U+0301, its bits are 0011 00 00 0001 meaning x=0x0c and y=0x01, so it is represented by the bytes 0xcc 0x81. Similarly,

Cyrillic “Á” → U+0410 → 0100 00 01 00000x10 0x100xd0 0x90.
Cyrillic “а́” → U+0430 → 0100 00 11 00000x10 0x200xd0 0xb0.

which disagrees with the dump output, but

Latin “Á” → U+00C1 → 0000 11 00 00010x03 0x010xc3 0x81.

So my saved Á á Ó ó É é were in fact Latin, as all six only use two bytes, so no combining accent!


Notes

1 I have asked about this in How can I type Russian characters with accents in KDE?. An answer to made me realise that I wanted the functionality described in this question.

2
  • FWIW, the Unicode standard has a whole technical annex about the underlying issues of combining accents, if you're curious (or have trouble falling asleep). – Ulrich Schwarz Jun 16 '20 at 18:24
  • @UlrichSchwarz: I am indeed curious, and sometimes have problems falling asleep, but I fear that a computer screen is more likely to be part of the problem than of a solution ;) I see that an impressive amount of effort has gone into this (and that IBM does not claim IPR, or appears not to!). – PJTraill Jun 16 '20 at 19:34
2

Emacs

C-x = (M-x what-cursor-position) shows code point information about the current character at the bottom of the screen, for example

Char: И (1048, #o2030, #x418, file ...) point=7 of 8 (75%) column=0

C-u C-x = opens a window with more information, including the code point, the byte representation, metadata about the Unicode character, what font is used to display the character, etc.

             position: 7 of 8 (75%), column: 0
            character: И (displayed as И) (codepoint 1048, #o2030, #x418)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x0418
               script: cyrillic
               syntax: w    which means: word
             category: .:Base, L:Left-to-right (strong), Y:2-byte Cyrillic, c:Chinese, h:Korean, j:Japanese, y:Cyrillic
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #xD0 #x98
            file code: #xD0 #x98 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-DAMA-Ubuntu Mono-normal-normal-normal-*-17-*-*-*-m-0-iso10646-1 (#x2CB)
         Unicode data:
                 Name: CYRILLIC CAPITAL LETTER I
             Category: Letter, Uppercase
      Combining class: Lu
        Bidi category: Lu
             Old name: CYRILLIC CAPITAL LETTER II
            Lowercase: и

Character code properties: customize what to show
  name: CYRILLIC CAPITAL LETTER I
  old-name: CYRILLIC CAPITAL LETTER II
  general-category: Lu (Letter, Uppercase)
  decomposition: (1048) ('И')

[back]

Command line

The unicode utility (available in some distributions such as Debian/Ubuntu/…, or with pip3 install unicode) shows information about one or more Unicode characters. Note that if you copy-paste from an editor, that editor may encode the clipboard differently from the file.

$ unicode И
U+0418 CYRILLIC CAPITAL LETTER I
UTF-8: d0 98 UTF-16BE: 0418 Decimal: И Octal: \02030
И (и)
Lowercase: 0438
Category: Lu (Letter, Uppercase)
Bidi: L (Left-to-Right)
6
  • The unicode command certainly looks very useful, but who or what and how trustworthy is kassiopeia.juls.savba.sk/~garabik ? – PJTraill Jun 16 '20 at 19:20
  • I have rather on-off feelings about emacs; this sounds like a reason to take another look ;) – PJTraill Jun 16 '20 at 19:21
  • @PJTraill This program is available as a package at least on Debian/Ubuntu/…. – Gilles 'SO- stop being evil' Jun 16 '20 at 19:34
  • I have just seen that the emacs commands work just as advertised, and the long form also shows how an а́ has been composed. – PJTraill Jun 16 '20 at 19:35
  • It does not seem to be in my repositories (Suse pacman, Mozilla), but perhaps I just have to snuffle around a bit more. – PJTraill Jun 16 '20 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.