Distinguish ascii from UTF-8 characters in the same file

Question

On Ubuntu 18.04, I created a dummy text file with just one UTF-8 character, è. The other characters are all ascii:

$ cat dummytext
Hello
Helloè

This is the resulting hexdump:

$ hexdump -C dummyfile
00000000  48 65 6c 6c 6f 0a 48 65  6c 6c 6f c3 a8 0a        |Hello.Hello...|
0000000e

The file is identified as

$ file dummyfile
dummyfile2: UTF-8 Unicode text

Each character is represented by a single byte, except for the UTF-8 è character, which is c3a8, so it is represented by 2 bytes. How can the file contents be correctly interpreted, if the number of bytes used to represent each character is not constant?

My guess: maybe the parser, when encountering a hex value which is greater than the last ascii character 7F (and this is the case of c3), is forced to read at least another byte, to determine the right character to be printed?

Answer 1

From the BSD manual, section 5, the page on UTF8 reads:

DESCRIPTION

The UTF-8 encoding represents UCS-4 characters as a sequence of octets, using between 1 and 6 for each character. It is backwards compatible with ASCII, so 0x00-0x7f refer to the ASCII character set.

The multibyte encoding of non-ASCII characters consist entirely of bytes whose high order bit is set. The actual encoding is represented by the following table:

 [0x00000000 - 0x0000007f] [00000000.0bbbbbbb] -> 0bbbbbbb
 [0x00000080 - 0x000007ff] [00000bbb.bbbbbbbb] -> 110bbbbb, 10bbbbbb
 [0x00000800 - 0x0000ffff] [bbbbbbbb.bbbbbbbb] ->
         1110bbbb, 10bbbbbb, 10bbbbbb
 [0x00010000 - 0x001fffff] [00000000.000bbbbb.bbbbbbbb.bbbbbbbb] ->
         11110bbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
 [0x00200000 - 0x03ffffff] [000000bb.bbbbbbbb.bbbbbbbb.bbbbbbbb] ->
         111110bb, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
 [0x04000000 - 0x7fffffff] [0bbbbbbb.bbbbbbbb.bbbbbbbb.bbbbbbbb] ->
         1111110b, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb

If more than a single representation of a value exists (for example, 0x00; 0xC0 0x80;0xE0 0x80 0x80), the shortest representation is always used. Longer ones are detected as an error as they pose a potential security risk, and destroy the 1:1 character:octet sequence mapping.

From the Linux manual, section 7, the page on UTF8 similarly reads:

DESCRIPTION

[... UTF-8 is situationally better than UCS-2 in part because i]n addition, the majority of UNIX tools expect ASCII files and can't read 16-bit words as characters without major modifications. [...]

The UTF-8 encoding of Unicode and UCS does not have these problems and is the common way in which Unicode is used on UNIX-style operating systems.

Properties

The UTF-8 encoding has the following nice properties:

• UCS characters 0x00000000 to 0x0000007f (the classic US-ASCII characters) are encoded simply as bytes 0x00 to 0x7f (ASCII compatibility). This means that files and strings which contain only 7-bit ASCII characters have the same encoding under both ASCII and UTF-8.

So it's not really possible to distinguish ASCII from UTF-8 because, in a UTF-8 file, ASCII is UTF-8. file looks at the first 96KiB of a file and tries to determine what it is. Because it sees more than zero UTF-8 code sequences, it determines the file to be UTF-8 because it is a strict superset of ASCII.