0

I have a text like the following:

[]përgjithshme [" është "] 28.748 [< kilometra katrorë.>]

This text contains cyrillic characters in the range U400 - U4FF.

I want to strip out the non-alphas such as: []"<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:

tr/a-z/ /cs;

This remove also the cyrillic chars above like ë.

Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?

2

tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)

You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.

Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:

$ echo '[]përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
 përgjithshme është kilometra katrorë $ 

$ echo '[]përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
 përgjithshme   është   28 748   kilometra katrorë 
$ 
  • Thanks. Didn't realize that you could specify utf-8 with the -C parameter. – bytebiscuit Dec 29 '18 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.