If the terminal emulator is well-designed and configured appropriately, it will ensure that the value of the environment variable
LC_CTYPE is set to a value that is consistent with its encoding. Unfortunately, in practice, checking
LC_CTYPE is not always reliable: it may be unset or wrong. (Other environment variables may convey the locale settings, see What should I set my locale to and what are the implications of doing so? for details.)
If you have some idea of which character encodings are likely, you may be able to determine the encoding via heuristics. Display a byte string that has a different width in different encodings, and find out by how much it makes the cursor move. This won't help you in all cases, for example it can't distinguish between single-byte encodings. But if for you the only two likely possibilities are UTF-8 and one legacy encoding, that works well. In my shell startup, I set
LC_CTYPE in this way, using a script
widthof which I posted in Get the display width of a string of characters.
widthof -1 displays a 4-byte string which represents 2 characters in UTF-8, and in which only 3 bytes are printable latin-N characters. Thus a width of 2 means UTF-8 (or some other multibyte encoding, which is not likely for me), a width of 3 means latin-N (with no way to know N), and 4 means some single-byte encoding with printable characters in the range 128–159.
case $? in
0) export LC_CTYPE=C;; # 7-bit charset
2) locale_search .utf8 .UTF-8;; # utf8
3) locale_search .iso88591 .ISO8859-1 .latin1 '';; # 8-bit with nonprintable 128-159, we assume latin1
4) locale_search .iso88591 .ISO8859-1 .latin1 '';; # some full 8-bit charset, we assume latin1
*) export LC_CTYPE=C;; # weird charset