5

I have a few shell scripts with UTF8-encoded characters inside and i want to be sure that they are decoded correctly regardless of the machine locale settings.

Is it possible to force the shell (bash or sh) to detect the correct script encoding? (something similar to the python or ruby encoding cookie)

The solution could be a custom shebang like:

#!/bin/bash --utf8

The solution should aim to better portability, so it is not necessary to stick with bash.

EDIT: maybe i've found a possible solution using a recursive script call:

# check if current locale is UTF8-based (otherwise this script may not work correctly)
locale | grep -q 'UTF-8'
if [ $? -ne 0 ]; then
    export LC_ALL=en_GB.UTF-8
    # recursive call this script with the modified environment
    $0 "$@"
    exit $?
fi
  • use the locale command to get a list of encodings available to you and put the one most suitable to you in LC_ALL for the duration of your special chars evaluation. – mikeserv Dec 19 '15 at 8:24
  • this can only be done BEFORE the script is launched, i want to force the encoding from within the script, to make it easier for the user. – eadmaster Dec 19 '15 at 8:46
  • 1
    yeah. so why do you think it can only happen before? – mikeserv Dec 19 '15 at 8:55
  • if i set LC_ALL inside the script, the shell won't re-decode the strings (i guess string decodings happens when the script is loaded). So, it can only be done before the shell istance is created. – eadmaster Dec 19 '15 at 12:14
  • 1
    that's not true. – mikeserv Dec 19 '15 at 18:13
2

Bash stores strings as byte strings, and performs operations according to the current LC_CTYPE setting. So there is no need to restart bash: just set the LC_CTYPE or LC_ALL variable to your desired locale. Note that if you store a string in a variable or function, what matters is the encoding at the time the variable is expanded or the relevant command in the function is executed. Here's a script that demonstrates that:

#!/bin/bash
LC_CTYPE=en_US.utf8
v_utf8='é'
n_utf8=${#v_utf8}
f_utf8 () { tmp='é'; echo ${#tmp}; }
echo "UTF-8 in UTF-8: $n_utf8 $(f_utf8)"
LC_CTYPE=en_US
v_latin1='é'
n_latin1=${#v_latin1}
f_latin1 () { tmp='é'; echo ${#tmp}; }
echo "Latin 1 in Latin 1: $n_latin1 $(f_latin1)"
echo "UTF-8 in Latin 1: ${#v_utf8} $(f_utf8)"
LC_CTYPE=en_US.utf8
echo "Latin 1 in UTF-8: ${#v_latin1} $(f_latin1)"

Output:

UTF-8 in UTF-8: 1 1
Latin 1 in Latin 1: 2 2
UTF-8 in Latin 1: 2 2
Latin 1 in UTF-8: 1 1

As you can see, the length of the string is calculated according to the current value of LC_CTYPE, regardless of the value at the time of definition.

  • ok, but can i assume most shells have the same behavior? (as i've said i am looking for the most portable/reliable solution) – eadmaster Dec 20 '15 at 6:59
  • @eadmaster I see the same behavior with bash (at least as far back to 3.1.17), ATT ksh (at least as far back to 93r) and zsh (at least as far back to 4.3.6). Dash (as of 0.5.7), posh (as of 0.12.3), mksh (as of 50d) and BusyBox ash (as of 1.22.0) don't support multibyte locales anyway. – Gilles 'SO- stop being evil' Dec 20 '15 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.