How is data encoded in pipes/STDOUT/STDIN?

Question

I have recently been looking into various text encodings and I am not able to find any good sources on how data is encoded in pipes.

Here are some general assumptions I have:

1. Pipes deal with binary, and are agnostic to the encoding
2. Applications on each side of the pipe (including STDOUT/STDIN) should have consensus on the text encoding format
3. The terminal/console also counts as one of these applications and should use the same encoding.
4. Unix applications default to UTF-8 but can be changed.

Are these accurate? Can someone extend on how these would work in a system with different defaults?

Follow on question:

• What do programs like cat send to the terminal? Do they "think" in unicode? Or do they just read bytes and send out bytes and it is up to the terminal to interpret the encoded text?

I have tried changing encodings in the terminal but it doesn't seem to help.

$ printf 'ö' | hexdump
0000000 c3 b6
0000002

$ export LANG=en_US.UTF-16
$ printf 'ö' | hexdump
0000000 c3 b6
0000002

Answer 1

I'll address each of your points below:

1. Pipes deal with binary, and are agnostic to the encoding
   Correct.

2. Applications on each side of the pipe (including STDOUT/STDIN) should have consensus on the text encoding format
   I wouldn't even go so far as to say text encoding; it doesn't need to be text (although it often is). The application reading from the pipe needs to know what to expect from the application writing to the pipe.

3. The terminal/console also counts as one of these applications and should use the same encoding
   The terminal isn't involved in the pipe. If we consider the case where a process' standard output is written to the terminal, then the terminal will interpret those bytes. That could be "text", or control codes that tell the terminal to do things like clear the screen or reposition the cursor.
   
   As an example, consider:

   $ clear | hexdump -c
   0000000 033   [   H 033   [   2   J 033   [   3   J
   

   That's <esc>[H<esc>[2J<esc>[3J; they're ANSI control sequences. The terminal translates that to clear the screen. See https://en.wikipedia.org/wiki/ANSI_escape_code for more information on that.

4. Unix applications default to UTF-8 but can be changed
   Again, this doesn't have anything directly to do with pipes. I think the the default is "C", which I believe is just the basic ASCII character set. The LANG environment variable generally controls the character encoding that programs use.