Why doesn't my Perl play nice with Unicode?

Question

On my new Arch installation, perl doesn't seem to play nice with Unicode. For example, given this input file:

ελα ρε
王小红

This command should give me the last two characters of each line:

$ perl -CIO -pe 's/.*(..)$/$1/' file
ε
º¢

However, as you can see above, I get gibberish. The correct output is:

ρε
小红

I know that my terminal (gnome-terminator) supports UTF-8 since these both work as expected:

$ cat file
ελα ρε
王小红
$ perl -pe '' file
ελα ρε
王小红

Unfortunately, without -CIO, perl doesn't deal with the files correctly either:

$ perl -pe 's/.*(..)$/$1/' file
ε
��

It also shouldn't be a locale issue:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

I'm guessing I need to install some Perl packages, but I don't know which ones. Some relevant information:

$ perl --version | grep subversion
This is perl 5, version 22, subversion 0 (v5.22.0) built for x86_64-linux-thread-multi

$ pacman -Qs unicode
local/fribidi 0.19.7-1
    A Free Implementation of the Unicode Bidirectional Algorithm
local/icu 55.1-1
    International Components for Unicode library
local/libunistring 0.9.6-1
    Library for manipulating Unicode strings and C strings
local/perl 5.22.0-1 (base)
    A highly capable, feature-rich programming language
local/perl-unicode-stringprep 1.105-1
    Preparation of Internationalized Strings (RFC 3454)
local/perl-unicode-utf8simple 1.06-5
    Conversions to/from UTF8 from/to characterse
local/ttf-arphic-uming 0.2.20080216.1-5
    CJK Unicode font Ming style

How can I get my perl installation to play nice with Unicode?

Answer 1

The issue you are describing is standard behaviour on the systems I tested on. I and O affect stdin and stdout, so this should work:

→ cat data | perl -CIO -pe 's/.*(..)$/$1/'
ρε
小红

Whereas this might not:

→ perl -CIO -pe 's/.*(..)$/$1/' data
ε
º¢

There are two more options to perl -C that produce your desired behaviour.

i     8   UTF-8 is the default PerlIO layer for input streams
o    16   UTF-8 is the default PerlIO layer for output streams

Which is basically saying to perl, use a file open form:

open(F, "<:utf8", "data");

or you can use perl -CSD which is shorthand for perl -CIOEio

S     7   I + O + E
D    24   i + o

Then you get

→ perl -CSD -pe 's/.*(..)$/$1/' data
ρε
小红

If the PERLIO environment variable is set and includes :utf8 this behaviour would also be enabled.

It looks like the default behaviour for perl isn't modifiable at configure/compile time either (cuonglm comment below). Arch certainly doesn't set anything. I doubt debian perl packages would modify default behaviour.

Answer 2

That's not a system issue but the perl itself.

-CIO only set UTF-8 encoding on STDIN and STDOUT, two of three perl predefined filehandle (you have -E for STDERR as well).

When you use:

perl -CIO -pe 's/.*(..)$/$1/' file

perl use diamond operator <> for processing file. Since when diamond operator <> used open (with two arguments form) to create new filehandle for each file from command line, these filehandle won't be affected by the UTF-8 encoding you set on STDIN and STDOUT.

So, you can pass the content of file to perl through its stdin, and it will work:

perl -CIO -pe 's/.*(..)$/$1/' <file

For other options see @Matt's answer.

---

In case you want perl use your locale for default encoding layer, you can use:

perl -Mopen=:locale -pe 's/.*(..)$/$1/' file

---

When you use PERLIO for setting the encoding layer, you should use :encoding(uf8) instead of :utf8.

Using :utf8 skip the encoding step, and can cause problem when reading invalid UTF-8 bytes sequences and leading to security problems.