6

How do I get perl to properly replace UTF-8 character from a shell?

The examples use stdin, but I need something that works for perl ... file too.

This is what I expect:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\\1]/g"
ABC[æ][ø][å]DEF

This is what I get:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\\1]/g"
ABCæøåDEF

Replacing the Unicode characters with ASCII works instantly:

$ echo ABC123DEF | perl -CS -pe "s/([123])/[\\1]/g"
ABC[1][2][3]DEF

My environment:

perl 5.18.2
Bash 3.2.57
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8

3 Answers 3

8

Use this :

 $ echo 'ABCæøåDEF' |
    perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'

Works also for files

Output :

ABC[æ][ø][å]DEF

Note :

  • substitutions: \\1 is for , \1 is for and in we use $1
  • check perldoc perlrun for -CSD utf8 tricks
8
  • Very nice! I know you can do export PERL_UNICODE=S to avoid -CS. Is there a similar thing you can do to avoid -Mutf8? alias perl="/usr/bin/perl -Mutf8" if nothing else? I always use UTF-8.
    – forthrin
    Apr 2, 2018 at 13:39
  • Please read this authoritative post stackoverflow.com/questions/6162484/… Apr 2, 2018 at 13:43
  • 1
    Instead of perl -CS -Mopen=":std,IN,:encoding(utf-8)", why not perl -CSD?
    – haukex
    Apr 2, 2018 at 17:45
  • 2
    @forthrin Re "perl is not meant for Unicode." Perl has excellent Unicode support, but for backwards compatibility it is not enabled by default, including for oneliners. If all you use Perl for is oneliners, then yes, you may have to jump through some hoops, but if you write scripts, then you'll have an easier time.
    – haukex
    Apr 2, 2018 at 17:45
  • 1
    What I need is a string replacement tool on the command line, because this is something I do very often. I can confirm that your suggestion works for both stdin and files. I really appreciate the help.
    – forthrin
    Apr 3, 2018 at 5:42
1

Your input:

$ cat input.txt 
ABCæøåDEF
$ hexdump -C input.txt 
00000000  41 42 43 c3 a6 c3 b8 c3  a5 44 45 46 0a           |ABC......DEF.|
0000000d

One good way IMO is the -C option plus utf8:

$ perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'
ABC[æ][ø][å]DEF

If you don't want to use UTF-8 on the command line, you can always write your Perl code in plain ASCII and use escapes such as \xAB, \x{ABCD}, or in newer Perls \N{U+ABCD} or \N{CHARNAME}:

$ perl -CSD -pe 's/([\xE6\xF8\xE5])/[$1]/g' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSD -pe 's/([\xE6\xF8\xE5])/[$1]/g'
ABC[æ][ø][å]DEF

This one is getting a little creative: @ARGV will be interpreted as UTF-8, so you can keep your source code as ASCII and pass the UTF-8 characters via a command line argument (not necessarily the nicest solution, just showing how you could make use of the the -CA option):

$ perl -CSDA -pe 'BEGIN{$p=shift;} s/($p)/[$1]/g' '[æøå]' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSDA -pe 'BEGIN{$p=shift;} s/($p)/[$1]/g' '[æøå]'
ABC[æ][ø][å]DEF

Or, of course you can always turn the oneliner into an actual script, where you can

use warnings;
use 5.012;
use utf8;
use open qw/:std :encoding(UTF-8)/;
use charnames qw/:full :short/;

Further reading: perlunitut, perlunifaq, perluniintro, perlunicode, perlunicook.

1
  • Appreciate the information about the alternative character syntaxes.
    – forthrin
    Apr 3, 2018 at 5:43
-2

$ echo 'ABCæøåDEF' | perl -CS -Mutf8 -pe 's/([æøå])/[$1]/g'

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .