I have recently been looking into various text encodings and I am not able to find any good sources on how data is encoded in pipes.
Here are some general assumptions I have:
- Pipes deal with binary, and are agnostic to the encoding
- Applications on each side of the pipe (including STDOUT/STDIN) should have consensus on the text encoding format
- The terminal/console also counts as one of these applications and should use the same encoding.
- Unix applications default to UTF-8 but can be changed.
Are these accurate? Can someone extend on how these would work in a system with different defaults?
Follow on question:
- What do programs like
catsend to the terminal? Do they "think" in unicode? Or do they just read bytes and send out bytes and it is up to the terminal to interpret the encoded text?
I have tried changing encodings in the terminal but it doesn't seem to help.
$ printf 'ö' | hexdump 0000000 c3 b6 0000002
$ export LANG=en_US.UTF-16 $ printf 'ö' | hexdump 0000000 c3 b6 0000002