2

I have recently been looking into various text encodings and I am not able to find any good sources on how data is encoded in pipes.

Here are some general assumptions I have:

  1. Pipes deal with binary, and are agnostic to the encoding
  2. Applications on each side of the pipe (including STDOUT/STDIN) should have consensus on the text encoding format
  3. The terminal/console also counts as one of these applications and should use the same encoding.
  4. Unix applications default to UTF-8 but can be changed.

Are these accurate? Can someone extend on how these would work in a system with different defaults?

Follow on question:

  • What do programs like cat send to the terminal? Do they "think" in unicode? Or do they just read bytes and send out bytes and it is up to the terminal to interpret the encoded text?

I have tried changing encodings in the terminal but it doesn't seem to help.

$ printf 'ö' | hexdump
0000000 c3 b6
0000002
$ export LANG=en_US.UTF-16
$ printf 'ö' | hexdump
0000000 c3 b6
0000002
  • I think that typically, the terminal uses the value of LANG that is present in the terminal's environment. export LANG=en_US.UTF-16 changes the shell's environment, not the terminal's. So, you'll need to set LANG for the terminal prior to running the terminal itself. – mpb Jul 3 at 0:34
7

I'll address each of your points below:

  1. Pipes deal with binary, and are agnostic to the encoding
    Correct.

  2. Applications on each side of the pipe (including STDOUT/STDIN) should have consensus on the text encoding format
    I wouldn't even go so far as to say text encoding; it doesn't need to be text (although it often is). The application reading from the pipe needs to know what to expect from the application writing to the pipe.

  3. The terminal/console also counts as one of these applications and should use the same encoding
    The terminal isn't involved in the pipe. If we consider the case where a process' standard output is written to the terminal, then the terminal will interpret those bytes. That could be "text", or control codes that tell the terminal to do things like clear the screen or reposition the cursor.

    As an example, consider:

    $ clear | hexdump -c
    0000000 033   [   H 033   [   2   J 033   [   3   J
    

    That's <esc>[H<esc>[2J<esc>[3J; they're ANSI control sequences. The terminal translates that to clear the screen. See https://en.wikipedia.org/wiki/ANSI_escape_code for more information on that.

  4. Unix applications default to UTF-8 but can be changed
    Again, this doesn't have anything directly to do with pipes. I think the the default is "C", which I believe is just the basic ASCII character set. The LANG environment variable generally controls the character encoding that programs use.

| improve this answer | |
  • 4
    Note that there are utilities like iconv that translate between different encodings. For example, ... | iconv -f ISO-8859-15 -t UTF-8 | ... will take in ISO-8859-15 from its left pipe, and output UTF-8 to its right one. – Gordon Davisson Jul 3 at 1:22
  • 2
    Also probably worth mentioning that while UTF-8 has become ubiquitous, for much of the history of Unix-like OSs the general assumption would have been 7-bit ASCII or (later on) an 8-bit extension (usually ISO Latin-1 AKA ISO 8859-1, or Windows Latin-1 AKA CP1252). (Those of us who have had to deal with the pain of conflicting character encodings welcome the prospect of UTF-8 becoming universal!) – gidds Jul 3 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.