Remove non-alphas while ignoring certain unicode chars

Question

I have a text like the following:

[]përgjithshme [" është "] 28.748 [< kilometra katrorë.>]

This text contains cyrillic characters in the range U400 - U4FF.

I want to strip out the non-alphas such as: []"<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:

tr/a-z/ /cs;

This remove also the cyrillic chars above like ë.

Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?

Answer 1

tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)

You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.

Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:

$ echo '[]përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
 përgjithshme është kilometra katrorë $ 

$ echo '[]përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
 përgjithshme   është   28 748   kilometra katrorë 
$