On my new Arch installation, perl doesn't seem to play nice with Unicode. For example, given this input file:

ελα ρε

This command should give me the last two characters of each line:

$ perl -CIO -pe 's/.*(..)$/$1/' file

However, as you can see above, I get gibberish. The correct output is:


I know that my terminal (gnome-terminator) supports UTF-8 since these both work as expected:

$ cat file
ελα ρε
$ perl -pe '' file
ελα ρε

Unfortunately, without -CIO, perl doesn't deal with the files correctly either:

$ perl -pe 's/.*(..)$/$1/' file

It also shouldn't be a locale issue:

$ locale

I'm guessing I need to install some Perl packages, but I don't know which ones. Some relevant information:

$ perl --version | grep subversion
This is perl 5, version 22, subversion 0 (v5.22.0) built for x86_64-linux-thread-multi

$ pacman -Qs unicode
local/fribidi 0.19.7-1
    A Free Implementation of the Unicode Bidirectional Algorithm
local/icu 55.1-1
    International Components for Unicode library
local/libunistring 0.9.6-1
    Library for manipulating Unicode strings and C strings
local/perl 5.22.0-1 (base)
    A highly capable, feature-rich programming language
local/perl-unicode-stringprep 1.105-1
    Preparation of Internationalized Strings (RFC 3454)
local/perl-unicode-utf8simple 1.06-5
    Conversions to/from UTF8 from/to characterse
local/ttf-arphic-uming 0.2.20080216.1-5
    CJK Unicode font Ming style

How can I get my perl installation to play nice with Unicode?

  • 2
    To handle unicode I am using utf8::all so your both should work: perl -Mutf8::all -pe 's/.*(..)$/$1/' file or perl -Mutf8::all -CIO -pe 's/.*(..)$/$1/' file.
    – taliezin
    Nov 5, 2015 at 18:31
  • @taliezin thanks, that helped. After installing the utf8::all module, the command you gave did indeed work as expected. I would still like to have the standard -CIO options working though. I shouldn't need to call another module for this.
    – terdon
    Nov 5, 2015 at 18:35
  • perldoc.perl.org/perlunicode.html -- this is one area where Perl is behind other languages. Nov 5, 2015 at 18:37
  • @glennjackman perhaps, but I know this sort of think used to work on my Debian and Choroba has confirmed that it works on his system. So, there should be a way to get it to work on my Arch as well.
    – terdon
    Nov 5, 2015 at 18:38

2 Answers 2


The issue you are describing is standard behaviour on the systems I tested on. I and O affect stdin and stdout, so this should work:

→ cat data | perl -CIO -pe 's/.*(..)$/$1/'

Whereas this might not:

→ perl -CIO -pe 's/.*(..)$/$1/' data

There are two more options to perl -C that produce your desired behaviour.

i     8   UTF-8 is the default PerlIO layer for input streams
o    16   UTF-8 is the default PerlIO layer for output streams

Which is basically saying to perl, use a file open form:

open(F, "<:utf8", "data");

or you can use perl -CSD which is shorthand for perl -CIOEio

S     7   I + O + E
D    24   i + o

Then you get

→ perl -CSD -pe 's/.*(..)$/$1/' data

If the PERLIO environment variable is set and includes :utf8 this behaviour would also be enabled.

It looks like the default behaviour for perl isn't modifiable at configure/compile time either (cuonglm comment below). Arch certainly doesn't set anything. I doubt debian perl packages would modify default behaviour.

  • What do you mean when saying default is easily modifiable at configure/compile time? AFAICT, the compile time only instruct perl to use perl own io abstraction instead of system stdio, it does mean you can set UTF-8 as default encoding layer for perl.
    – cuonglm
    Nov 6, 2015 at 3:17
  • Well, I said I'm not sure if the default was modifiable or not. I was trying to think of reasons his arch system would be different to a debian system. Did you mean to write "it doesn't mean" in your last sentence?
    – Matt
    Nov 6, 2015 at 6:01
  • Yes, I mean "it does not mean", my bad typo.
    – cuonglm
    Nov 6, 2015 at 6:22
  • Thnx. It's been a while since I've needed to compile perl =)
    – Matt
    Nov 6, 2015 at 6:35

That's not a system issue but the perl itself.

-CIO only set UTF-8 encoding on STDIN and STDOUT, two of three perl predefined filehandle (you have -E for STDERR as well).

When you use:

perl -CIO -pe 's/.*(..)$/$1/' file

perl use diamond operator <> for processing file. Since when diamond operator <> used open (with two arguments form) to create new filehandle for each file from command line, these filehandle won't be affected by the UTF-8 encoding you set on STDIN and STDOUT.

So, you can pass the content of file to perl through its stdin, and it will work:

perl -CIO -pe 's/.*(..)$/$1/' <file

For other options see @Matt's answer.

In case you want perl use your locale for default encoding layer, you can use:

perl -Mopen=:locale -pe 's/.*(..)$/$1/' file

When you use PERLIO for setting the encoding layer, you should use :encoding(uf8) instead of :utf8.

Using :utf8 skip the encoding step, and can cause problem when reading invalid UTF-8 bytes sequences and leading to security problems.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .