6

How do I get perl to properly replace UTF-8 character from a shell?

The examples use stdin, but I need something that works for perl ... file too.

This is what I expect:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\\1]/g"
ABC[æ][ø][å]DEF

This is what I get:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\\1]/g"
ABCæøåDEF

Replacing the Unicode characters with ASCII works instantly:

$ echo ABC123DEF | perl -CS -pe "s/([123])/[\\1]/g"
ABC[1][2][3]DEF

My environment:

perl 5.18.2
Bash 3.2.57
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
8

Use this :

 $ echo 'ABCæøåDEF' |
    perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'

Works also for files

Output :

ABC[æ][ø][å]DEF

Note :

  • substitutions: \\1 is for , \1 is for and in we use $1
  • check perldoc perlrun for -CSD utf8 tricks
  • Very nice! I know you can do export PERL_UNICODE=S to avoid -CS. Is there a similar thing you can do to avoid -Mutf8? alias perl="/usr/bin/perl -Mutf8" if nothing else? I always use UTF-8. – forthrin Apr 2 '18 at 13:39
  • Please read this authoritative post stackoverflow.com/questions/6162484/… – Gilles Quenot Apr 2 '18 at 13:43
  • 1
    Instead of perl -CS -Mopen=":std,IN,:encoding(utf-8)", why not perl -CSD? – haukex Apr 2 '18 at 17:45
  • 2
    @forthrin Re "perl is not meant for Unicode." Perl has excellent Unicode support, but for backwards compatibility it is not enabled by default, including for oneliners. If all you use Perl for is oneliners, then yes, you may have to jump through some hoops, but if you write scripts, then you'll have an easier time. – haukex Apr 2 '18 at 17:45
  • 1
    What I need is a string replacement tool on the command line, because this is something I do very often. I can confirm that your suggestion works for both stdin and files. I really appreciate the help. – forthrin Apr 3 '18 at 5:42
1

Your input:

$ cat input.txt 
ABCæøåDEF
$ hexdump -C input.txt 
00000000  41 42 43 c3 a6 c3 b8 c3  a5 44 45 46 0a           |ABC......DEF.|
0000000d

One good way IMO is the -C option plus utf8:

$ perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'
ABC[æ][ø][å]DEF

If you don't want to use UTF-8 on the command line, you can always write your Perl code in plain ASCII and use escapes such as \xAB, \x{ABCD}, or in newer Perls \N{U+ABCD} or \N{CHARNAME}:

$ perl -CSD -pe 's/([\xE6\xF8\xE5])/[$1]/g' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSD -pe 's/([\xE6\xF8\xE5])/[$1]/g'
ABC[æ][ø][å]DEF

This one is getting a little creative: @ARGV will be interpreted as UTF-8, so you can keep your source code as ASCII and pass the UTF-8 characters via a command line argument (not necessarily the nicest solution, just showing how you could make use of the the -CA option):

$ perl -CSDA -pe 'BEGIN{$p=shift;} s/($p)/[$1]/g' '[æøå]' input.txt 
ABC[æ][ø][å]DEF
$ cat input.txt | perl -CSDA -pe 'BEGIN{$p=shift;} s/($p)/[$1]/g' '[æøå]'
ABC[æ][ø][å]DEF

Or, of course you can always turn the oneliner into an actual script, where you can

use warnings;
use 5.012;
use utf8;
use open qw/:std :encoding(UTF-8)/;
use charnames qw/:full :short/;

Further reading: perlunitut, perlunifaq, perluniintro, perlunicode, perlunicook.

  • Appreciate the information about the alternative character syntaxes. – forthrin Apr 3 '18 at 5:43
-2

$ echo 'ABCæøåDEF' | perl -CS -Mutf8 -pe 's/([æøå])/[$1]/g'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.