I have recently been looking into various text encodings and I am not able to find any good sources on how data is encoded in pipes.
Here are some general assumptions I have:
- Pipes deal with binary, and are agnostic to the encoding
- Applications on each side of the pipe (including STDOUT/STDIN) should have consensus on the text encoding format
- The terminal/console also counts as one of these applications and should use the same encoding.
- Unix applications default to UTF-8 but can be changed.
Are these accurate? Can someone extend on how these would work in a system with different defaults?
Follow on question:
- What do programs like
cat
send to the terminal? Do they "think" in unicode? Or do they just read bytes and send out bytes and it is up to the terminal to interpret the encoded text?
I have tried changing encodings in the terminal but it doesn't seem to help.
$ printf 'ö' | hexdump
0000000 c3 b6
0000002
$ export LANG=en_US.UTF-16
$ printf 'ö' | hexdump
0000000 c3 b6
0000002
LANG
that is present in the terminal's environment.export LANG=en_US.UTF-16
changes the shell's environment, not the terminal's. So, you'll need to setLANG
for the terminal prior to running the terminal itself.