From the BSD manual, section 5, the page on UTF8 reads:
DESCRIPTION
The UTF-8 encoding represents UCS-4 characters as a sequence of octets, using between 1 and 6 for each character. It is backwards
compatible with ASCII, so 0x00
-0x7f
refer to the ASCII character set.
The multibyte encoding of non-ASCII characters consist entirely of bytes whose high order bit is set. The actual encoding is
represented by the following table:
[0x00000000 - 0x0000007f] [00000000.0bbbbbbb] -> 0bbbbbbb
[0x00000080 - 0x000007ff] [00000bbb.bbbbbbbb] -> 110bbbbb, 10bbbbbb
[0x00000800 - 0x0000ffff] [bbbbbbbb.bbbbbbbb] ->
1110bbbb, 10bbbbbb, 10bbbbbb
[0x00010000 - 0x001fffff] [00000000.000bbbbb.bbbbbbbb.bbbbbbbb] ->
11110bbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
[0x00200000 - 0x03ffffff] [000000bb.bbbbbbbb.bbbbbbbb.bbbbbbbb] ->
111110bb, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
[0x04000000 - 0x7fffffff] [0bbbbbbb.bbbbbbbb.bbbbbbbb.bbbbbbbb] ->
1111110b, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
If more than a single representation of a value exists (for example, 0x00
; 0xC0 0x80
;0xE0 0x80 0x80
), the shortest representation
is always used. Longer ones are detected as an error as they pose a
potential security risk, and destroy the 1:1 character:octet sequence mapping.
From the Linux manual, section 7, the page on UTF8 similarly reads:
DESCRIPTION
[... UTF-8 is situationally better than UCS-2 in part because i]n addition, the majority of UNIX tools expect ASCII files and can't read 16-bit words as characters without major modifications. [...]
The UTF-8 encoding of Unicode and UCS does not have these problems and is the common way in which Unicode is used on UNIX-style operating systems.
Properties
The UTF-8 encoding has the following nice properties:
- UCS characters
0x00000000
to 0x0000007f
(the classic US-ASCII characters) are encoded simply as bytes 0x00
to 0x7f
(ASCII compatibility). This means that files and strings which contain only 7-bit ASCII characters have the same encoding under both ASCII and UTF-8.
So it's not really possible to distinguish ASCII from UTF-8 because, in a UTF-8 file, ASCII is UTF-8. file
looks at the first 96KiB of a file and tries to determine what it is. Because it sees more than zero UTF-8 code sequences, it determines the file to be UTF-8 because it is a strict superset of ASCII.
file
know that this is UTF-8, when it could instead be an old 8-bit encoding? followed by How does a UTF-8 decoder know where multiple-byte sequences begin and end?.file
as a further verification). DopeGhoti's answer fits to the second one. For the first one, maybefile
looks for bytes "whose high order bit is set" and then is able to guess if there is an UTF-8 encoding.