Replace UTF-8 characters with shell perl

Question

How do I get perl to properly replace UTF-8 character from a shell?

The examples use stdin, but I need something that works for perl ... file too.

This is what I expect:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\\1]/g"
ABC[æ][ø][å]DEF

This is what I get:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\\1]/g"
ABCæøåDEF

Replacing the Unicode characters with ASCII works instantly:

$ echo ABC123DEF | perl -CS -pe "s/([123])/[\\1]/g"
ABC[1][2][3]DEF

My environment:

perl 5.18.2
Bash 3.2.57
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8

Answer 1

Use this :

 $ echo 'ABCæøåDEF' |
    perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'

Works also for files

Output :

ABC[æ][ø][å]DEF

Note :

• substitutions: \\1 is for awk, \1 is for sed and in perl we use $1
• check perldoc perlrun for -CSD utf8 tricks

Answer 2

Your input:

$ cat input.txt 
ABCæøåDEF
$ hexdump -C input.txt 
00000000  41 42 43 c3 a6 c3 b8 c3  a5 44 45 46 0a           |ABC......DEF.|
0000000d

One good way IMO is the -C option plus utf8:

$ perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'
ABC[æ][ø][å]DEF

If you don't want to use UTF-8 on the command line, you can always write your Perl code in plain ASCII and use escapes such as \xAB, \x{ABCD}, or in newer Perls \N{U+ABCD} or \N{CHARNAME}:

$ perl -CSD -pe 's/([\xE6\xF8\xE5])/[$1]/g' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSD -pe 's/([\xE6\xF8\xE5])/[$1]/g'
ABC[æ][ø][å]DEF

This one is getting a little creative: @ARGV will be interpreted as UTF-8, so you can keep your source code as ASCII and pass the UTF-8 characters via a command line argument (not necessarily the nicest solution, just showing how you could make use of the the -CA option):

$ perl -CSDA -pe 'BEGIN{$p=shift;} s/($p)/[$1]/g' '[æøå]' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSDA -pe 'BEGIN{$p=shift;} s/($p)/[$1]/g' '[æøå]'
ABC[æ][ø][å]DEF

Or, of course you can always turn the oneliner into an actual script, where you can

use warnings;
use 5.012;
use utf8;
use open qw/:std :encoding(UTF-8)/;
use charnames qw/:full :short/;

Further reading: perlunitut, perlunifaq, perluniintro, perlunicode, perlunicook.

Answer 3

$ echo 'ABCæøåDEF' | perl -CS -Mutf8 -pe 's/([æøå])/[$1]/g'