3

I was working on a keymap script (map keys from one language keyboard layout to another). And after a lot of hard time trying to get everything working I found out that different characters are treated differently in all programs (perl, python). Then I run a simple test script (now simplified) in a terminal (kitty, gnome-terminal — it doesn't matter):

python -c 'import sys;print(len(sys.argv[1]))' テスト

And got an expected result:

3

But if I run this in a sh/bash (unix&utf-8) file:

#!/usr/bin/env bash
# or
#!/bin/sh
python -c 'import sys;print(len(sys.argv[1]))' テスト

I get (./test.sh):

9

And that's the reason all this encode/decode/upgrade/downgrade UTF-8 stuff didn't work in Perl (if I would run the command manually from terminal it would probably work without all this additional encoding functions).

Now I have a problem: why the exact same command gives me different results depending on the execution environment (terminal emulator vs shell script)? How can I fix this?

Update:

I forgot about my:

alias python='python3'

So with Python, running python3 explicitly makes everything work the same in both cases. But with Perl on the other hand:

echo 'print length $ARGV[0];' | perl -l -- - テスト

This works the same, but in both cases it outputs 9. With Perl there are no different versions and mine is 5.30.0 (which is printed in both cases exactly the same). Do I have to add some code in Perl itself to make it work like Python3 (length of 1 Unicode character is 1 and not 1-3 bytes)?

1 Answer 1

5

This isn't about the shell but about python. I can reproduce this by explicitly running the same command using python3 and then python2:

$ python3 -c 'import sys;print(len(sys.argv[1]))' テスト
3
$ python2 -c 'import sys;print(len(sys.argv[1]))' テスト
9

Since you are not using a full path to a specific python executable, both your terminal and your script will just take the first python they find in a directory listed in PATH. In your case, the PATH in your non-interactive shell (the one that runs the script) is not the same as the PATH in your interactive shells (in the terminal) and in the former, python apparently points to a Python2 executable.

I don't know why you have this, I would need to know much more about your setup and what operating system you are using, but a simple solution, assuming you are on a system that provides this, is to call the script with python3 instead of python:

python3 -c 'import sys;print(len(sys.argv[1]))' テスト

Alternatively, use the full path (see type -a python):

/usr/bin/python -c 'import sys;print(len(sys.argv[1]))' テスト

This way your results will always be consistent.

3
  • Thank you, I forgot about my alias (it's so convenient but not in scripts). Please read the updated question. I still don't get why Perl treats arguments differently (string is consists of bytes, therefore the length is greater than in Python3). I'm not familiar with all the things in Perl, so maybe I'm missing something.
    – Andrew15_5
    2 days ago
  • 2
    @Andrew15_5 I can just about guarantee you are missing something since i) Perl is actually very good at handling utf-8 and ii) handling utf-8 is ridiculously complicated so we all miss something unless we're actual experts. That said, in this case, a simple -CA (which tells perl its @ARGV array contains UTF-8 text) should be enough: perl -CA -le 'print length($ARGV[0])' テスト.
    – terdon
    2 days ago
  • I also found out about -CS which treats std(in|out|err) as UTF-8 (source: perldoc perlrun). Thank you for your answers, very helpful! Now I completed my script with a snap of a finger.
    – Andrew15_5
    2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.