4

On my new Arch installation, perl doesn't seem to play nice with Unicode. For example, given this input file:

ελα ρε
王小红

This command should give me the last two characters of each line:

$ perl -CIO -pe 's/.*(..)$/$1/' file
ε
º¢

However, as you can see above, I get gibberish. The correct output is:

ρε
小红

I know that my terminal (gnome-terminator) supports UTF-8 since these both work as expected:

$ cat file
ελα ρε
王小红
$ perl -pe '' file
ελα ρε
王小红

Unfortunately, without -CIO, perl doesn't deal with the files correctly either:

$ perl -pe 's/.*(..)$/$1/' file
ε
��

It also shouldn't be a locale issue:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

I'm guessing I need to install some Perl packages, but I don't know which ones. Some relevant information:

$ perl --version | grep subversion
This is perl 5, version 22, subversion 0 (v5.22.0) built for x86_64-linux-thread-multi

$ pacman -Qs unicode
local/fribidi 0.19.7-1
    A Free Implementation of the Unicode Bidirectional Algorithm
local/icu 55.1-1
    International Components for Unicode library
local/libunistring 0.9.6-1
    Library for manipulating Unicode strings and C strings
local/perl 5.22.0-1 (base)
    A highly capable, feature-rich programming language
local/perl-unicode-stringprep 1.105-1
    Preparation of Internationalized Strings (RFC 3454)
local/perl-unicode-utf8simple 1.06-5
    Conversions to/from UTF8 from/to characterse
local/ttf-arphic-uming 0.2.20080216.1-5
    CJK Unicode font Ming style

How can I get my perl installation to play nice with Unicode?

  • 2
    To handle unicode I am using utf8::all so your both should work: perl -Mutf8::all -pe 's/.*(..)$/$1/' file or perl -Mutf8::all -CIO -pe 's/.*(..)$/$1/' file. – taliezin Nov 5 '15 at 18:31
  • @taliezin thanks, that helped. After installing the utf8::all module, the command you gave did indeed work as expected. I would still like to have the standard -CIO options working though. I shouldn't need to call another module for this. – terdon Nov 5 '15 at 18:35
  • perldoc.perl.org/perlunicode.html -- this is one area where Perl is behind other languages. – glenn jackman Nov 5 '15 at 18:37
  • @glennjackman perhaps, but I know this sort of think used to work on my Debian and Choroba has confirmed that it works on his system. So, there should be a way to get it to work on my Arch as well. – terdon Nov 5 '15 at 18:38
6

The issue you are describing is standard behaviour on the systems I tested on. I and O affect stdin and stdout, so this should work:

→ cat data | perl -CIO -pe 's/.*(..)$/$1/'
ρε
小红

Whereas this might not:

→ perl -CIO -pe 's/.*(..)$/$1/' data
ε
º¢

There are two more options to perl -C that produce your desired behaviour.

i     8   UTF-8 is the default PerlIO layer for input streams
o    16   UTF-8 is the default PerlIO layer for output streams

Which is basically saying to perl, use a file open form:

open(F, "<:utf8", "data");

or you can use perl -CSD which is shorthand for perl -CIOEio

S     7   I + O + E
D    24   i + o

Then you get

→ perl -CSD -pe 's/.*(..)$/$1/' data
ρε
小红

If the PERLIO environment variable is set and includes :utf8 this behaviour would also be enabled.

It looks like the default behaviour for perl isn't modifiable at configure/compile time either (cuonglm comment below). Arch certainly doesn't set anything. I doubt debian perl packages would modify default behaviour.

  • What do you mean when saying default is easily modifiable at configure/compile time? AFAICT, the compile time only instruct perl to use perl own io abstraction instead of system stdio, it does mean you can set UTF-8 as default encoding layer for perl. – cuonglm Nov 6 '15 at 3:17
  • Well, I said I'm not sure if the default was modifiable or not. I was trying to think of reasons his arch system would be different to a debian system. Did you mean to write "it doesn't mean" in your last sentence? – Matt Nov 6 '15 at 6:01
  • Yes, I mean "it does not mean", my bad typo. – cuonglm Nov 6 '15 at 6:22
  • Thnx. It's been a while since I've needed to compile perl =) – Matt Nov 6 '15 at 6:35
2

That's not a system issue but the perl itself.

-CIO only set UTF-8 encoding on STDIN and STDOUT, two of three perl predefined filehandle (you have -E for STDERR as well).

When you use:

perl -CIO -pe 's/.*(..)$/$1/' file

perl use diamond operator <> for processing file. Since when diamond operator <> used open (with two arguments form) to create new filehandle for each file from command line, these filehandle won't be affected by the UTF-8 encoding you set on STDIN and STDOUT.

So, you can pass the content of file to perl through its stdin, and it will work:

perl -CIO -pe 's/.*(..)$/$1/' <file

For other options see @Matt's answer.


In case you want perl use your locale for default encoding layer, you can use:

perl -Mopen=:locale -pe 's/.*(..)$/$1/' file

When you use PERLIO for setting the encoding layer, you should use :encoding(uf8) instead of :utf8.

Using :utf8 skip the encoding step, and can cause problem when reading invalid UTF-8 bytes sequences and leading to security problems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.