Different encoding/Unicode interpretation using terminal vs using shell script

Question

I was working on a keymap script (map keys from one language keyboard layout to another). And after a lot of hard time trying to get everything working I found out that different characters are treated differently in all programs (perl, python). Then I run a simple test script (now simplified) in a terminal (kitty, gnome-terminal — it doesn't matter):

python -c 'import sys;print(len(sys.argv[1]))' テスト

And got an expected result:

3

But if I run this in a sh/bash (unix&utf-8) file:

#!/usr/bin/env bash
# or
#!/bin/sh
python -c 'import sys;print(len(sys.argv[1]))' テスト

I get (./test.sh):

9

And that's the reason all this encode/decode/upgrade/downgrade UTF-8 stuff didn't work in Perl (if I would run the command manually from terminal it would probably work without all this additional encoding functions).

Now I have a problem: why the exact same command gives me different results depending on the execution environment (terminal emulator vs shell script)? How can I fix this?

Update:

I forgot about my:

alias python='python3'

So with Python, running python3 explicitly makes everything work the same in both cases. But with Perl on the other hand:

echo 'print length $ARGV[0];' | perl -l -- - テスト

This works the same, but in both cases it outputs 9. With Perl there are no different versions and mine is 5.30.0 (which is printed in both cases exactly the same). Do I have to add some code in Perl itself to make it work like Python3 (length of 1 Unicode character is 1 and not 1-3 bytes)?

Answer 1

This isn't about the shell but about python. I can reproduce this by explicitly running the same command using python3 and then python2:

$ python3 -c 'import sys;print(len(sys.argv[1]))' テスト
3
$ python2 -c 'import sys;print(len(sys.argv[1]))' テスト
9

Since you are not using a full path to a specific python executable, both your terminal and your script will just take the first python they find in a directory listed in PATH. In your case, the PATH in your non-interactive shell (the one that runs the script) is not the same as the PATH in your interactive shells (in the terminal) and in the former, python apparently points to a Python2 executable.

I don't know why you have this, I would need to know much more about your setup and what operating system you are using, but a simple solution, assuming you are on a system that provides this, is to call the script with python3 instead of python:

python3 -c 'import sys;print(len(sys.argv[1]))' テスト

Alternatively, use the full path (see type -a python):

/usr/bin/python -c 'import sys;print(len(sys.argv[1]))' テスト

This way your results will always be consistent.