4

I understand that any character is comprised of one or more byte/s.

If I am not mistaken, at least in *nix operating systems, a character will generally (or totally?) be comprised of only one byte.

What is the difference between a byte and a character (at least *nixwise)?

4
  • 4
    Byte and Character were more or less synonyms when *nix was first invented, and the USA hadn't yet noticed that the entire world didn't use the English alphabet for writing.
    – alephzero
    Feb 24 at 3:05
  • 2
    Cross-site duplicate of the more complete question What's the difference between a character, a code point, a glyph and a grapheme? Feb 24 at 3:28
  • @alephzero had nothing to do with ignoring the rest of the world, and everything to do with the fact that data did very little travelling between computer systems, so it worked out fine to have many different mappings between 8-bit (or smaller) numbers and characters, one for each language group. Made much more efficient use of limited resources. The data tables necessary to answer a question like "is this an upper case letter?" for 2021 Unicode are bigger than the whole memory of many pre-1990 computers.
    – hobbs
    Feb 24 at 6:23
  • @alephzero When RAM costs hundreds of dollars per kilobyte, practical concerns dominate. It wasn't a culture war - memory was just hellishly expensive and it was ridiculous to spend double to support needs that nobody had. Today we think nothing of downloading 50MB just to see a cat act like a retard, but in 1973 it would have been unthinkable to suggest a Unicode style encoding scheme. You would need an entire storage device bigger than a person just to store a single font that actually made use of it, to say nothing of the huge compute cost of hauling those extra bytes around.
    – J...
    Feb 24 at 14:27
10

POSIXly, emphasis mine:

3.87 Character
A sequence of one or more bytes representing a single graphic symbol or control code.

In practice, the exact meaning depends on the locale in effect, e.g. under the "C" locale, printf '\xc3\xa4\xc3\xb6' |wc -m gives 4, since it effectively counts bytes; while under a UTF-8 locale that gives 2, since that's the two UTF-8 encoded characters äö. Assuming your terminal is also set to UTF-8, you could of course just write printf 'äö'.

(Note that wc -c is defined to count bytes, not characters, confusingly enough.)

Worse, character support also depends on the utility, and not everything deals with multi-byte characters cleanly (let alone all the quirks of Unicode). E.g. GNU tr deals with bytes, regardless of what its man page says:

$ printf ä | tr ä xy; echo
xx
$ printf ö | tr ä xy; echo
x�

The first is same as tr '\303\244' 'xy', so both bytes of ä get replaced, and that second happens because the first byte of both ä and ö is the same. Of course, if it really dealt with characters, those should print x and ö.

2
  • I never understood why those tools deal in bytes or characters. Almost always what you want is grapheme clusters, not bytes, characters, or code points. It's 2021 and a significant portion of tools on my 2021 macOS or my Ubuntu 20.10 still can't deal with my first name properly. Feb 24 at 13:20
  • @JörgWMittag, yes, the thing about combining characters is another thing entirely. But with a UTF-8 locale (even en_US.UTF-8), at least the fixed ö (as opposed to o + combining diaresis) seems to work with the usual text tools, not that I use them a lot. I'm lucky not to have non-ASCII characters in my name, and try to avoid them in e.g. filenames anyway (call it inertia or being over-careful, whatever). But e.g. grep 'J..g' does match the output of printf "J\xc3\xb6rg\n" for me, though not the output of printf "Jo\xcc\x88rg\n".
    – ilkkachu
    Feb 24 at 14:21
3

A byte is by convention and POSIX definition eight bits. A bit is a binary digit (i. e. the fundamental 1 or 0 that is at the base of nearly all digital computing).

A character is often one byte and in some contexts (e. g. ASCII) can be defined to be one byte in length. However, Unicode and UTF-8 and UTF-16 define expanded character sets wherein a single character (or glyph) can be defined by data payloads longer than one byte in length.

The single character:

Q̴̢̪̘̳̣̞̩̪̑̍̉̆̉͛̑̂̕͝

is a single character, but it is composed in Unicode by applying multiple accents (or diacritics) to the base glyph, the simple Q. This encoding is many more bytes than one in length: Putting solely that character into a file and displaying the contents with hexdump rather than cat on my locale yields:

$ hexdump -C demo
00000000  51 cc b4 cc 91 cc 8d cc  89 cc 86 cc 89 cd 9d cd  |Q...............|
00000010  9b cc 91 cc 95 cc 82 cc  aa cc 98 cc b3 cc a3 cc  |................|
00000020  a2 cc 9e cc a9 cc aa 0a                           |........|
00000028
8
  • 3
    I'll note: In many contexts, the Zalgo text here is considered multiple "characters" (really, multiple code points, "character" is too overloaded a term). It's a single grapheme, composed of multiple combining code points. Feb 24 at 3:30
  • 6
    "A byte is by definition eight bits", whos definition, and since when?
    – pipe
    Feb 24 at 4:42
  • 5
    "A byte is by definition eight bits." - According to Wikipedia, bytes on modern systems are 8 bits, but they are not 8 bits by definition: "The size of the byte has historically been hardware-dependent and no definitive standards existed that mandated the size. Sizes from 1 to 48 bits have been used. [...] The modern de facto standard of eight bits ..."
    – marcelm
    Feb 24 at 8:35
  • 1
    Also, it's not true that bits are "at the base of all digital computing". There are, for example, ternary computers that are based on three distinct states.
    – Psychonaut
    Feb 24 at 8:36
  • “Putting solely that character into a file” glosses over the difference between Unicode code points (of which that uses 20: U+0051 LATIN CAPITAL LETTER Q, U+0334 COMBINING TILDE OVERLAY,…) and bytes (to which the code points are encoded).  In this case you've used UTF-8 (which is by far the most common encoding), but others are possible.
    – gidds
    Feb 24 at 9:48
2

A byte is the basic element, usually 8 bits long (also called an octet), though there have been (and there probably still are) other sizes. With an 8-bit byte, you can encode 256 different values (from 0 to 255).

For characters, things vary based on the encoding and character set used.

  • The simplest most common encoding/character set is ASCII. Each character uses one byte (actually less, only 7 bits). It includes lowercase and uppercase letters of the english alphabet without diacritics (accents and the like), digits, common punctuation, and control characters.

  • Then we have a series of 8-bit character sets such as the ISO-8859 series, MS-DOS and Windows codepages, Mac charsets, etc.

    Those are supersets of ASCII (the first 128 values are the same as ASCII), with the 128 other values used for locale-specific characters (accented characters, alternative scripts such as greek or cyrillic...).

    This caused all sorts of headaches when transferring files between computers and even between programs, as not all used the same character set.

    In that case, one character was still one byte.

  • Then came the Unicode family, which tried to unify everything in a single set, which was obviously larger than 256, so didn't fit in a single byte.

  • At first, it was thought that 16 bits would be enough, and UCS-2 was devised, using 2 bytes per character (which would mean a maximum of 65536 possible characters, though not all were assigned, which allowed UTF-16).

  • Then it became clear that 2 bytes would not always be enough, so UTF-16 was introduced, which uses surrogate pairs to encode additional characters. For characters in the BMP (Basic Multilingual Plane), they still use 2 bytes exactly, but for the "extra" characters, they use 2 code units of 2 bytes each, for a total of 4 bytes.

    UTF-16 is the native encoding of Windows NT and successors. However even UTF-16 had issues, as not everybody agreed on the order of the two bytes: little-endian or big-endian, so we have UTF-16LE and UTF-16BE. With or without BOM.

  • There's also UCS-4 and UTF-32 which use 4 bytes per character (UTF-32 is limited to the values which can be expressed as UTF-16), but those are quite rare.

  • UTF-8 is a variable length encoding which is probably becoming the most common encoding. A character can be encoded as anywhere between 1 and 4 bytes.

    The genius in UTF-8 is that the ASCII part of Unicode (code points 0 to 127) is still encoded as a single byte, and code points beyond that are guaranteed to never include bytes between 0 and 127. This allows a certain level of compatibility with software where specific characters have a special meaning, including / (or \ or :) for paths, a lot of punctuation (!=+-*/^"'<>[]{} etc.) for programming languages and shells, control characters such as CR, LF or tab, spaces, etc.

But in Unicode, there's an additional complexity: code points can be composed. You can encode é as a single character é (U+00E9 LATIN SMALL LETTER E WITH ACUTE), or as e (U+0065 LATIN SMALL LETTER E) followed by ◌́ (U+0301 COMBINING ACUTE ACCENT). As shown in DopeGhoti's answer, you can stack up quite a few combining marks on a single letter!

Diacritics are not the only combining code points. There are many which are used to make variations, especially for emoji. You can change their skin colour (👨👨🏻👨🏼👨🏽👨🏾👨🏿), their gender (👨👩), their age (👩👧), combine them to make a family: 👨‍👩‍👧... The last one is 5 code points and takes 18 bytes!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.