3

Problem: I have text file with approximately 1M lines, each line consists of multiple words Some lines contain Russian or Arabic words (lets call them "bad lines"), I want to remove only these bad lines. The others are "good lines".

Conditions:

  • There are non-ascii characters in some of good lines, so just removing everything non-ascii doesn't solve the problem.
  • Each line is either completely bad or good, so this actually makes things a little bit easier.

So the solution that came to my mind is to just:

sort file.txt > sorted.txt

Anything starting with Russian or Arabic will be listed at the end. And then I will manually check the line they begin and then do

head -n X sorted.txt > clean.txt

to get rid of them. Is there a more elegant way to do this? What other Unix tools can be used to accomplish this task.

Sample input:

kedi
cat
кошка
القط
candy
şeker
конфеты
كاندي
çağrı
resumé

Desired output:

kedi
cat
candy
şeker
çağrı
resumé
2

This should work at least for your example:

$ perl -CS -Mutf8 -lne 's{ 
            (?= [\p{Arabic}\p{Cyrillic}] ) 
            [\p{Arabic}\p{Cyrillic}\p{Common}\p{Inherited}] + 
            (?<= [\p{Arabic}\p{Cyrillic}] ) }{}xg || print' < file

kedi
cat
candy
şeker
çağrı
resumé

The basic idea is to use the \p to define a set of code points, in this case Arabic or Cyrillic and if a line matches, it will not be printed. Unfortunately, I don't understand the details myself, this was provided by @tchrist in chat. I suggest you read his description there. I don't have time to decipher the regex now but I will add an explanation as soon as I get a chance to understand it myself. Given the source, I am assuming it is more complete and safer than my minimalist version below.


A simplified version of this that falls within my own understanding is

$ perl  -CS -ne '/[\p{Arabic}\p{Cyrillic}]+/ || print' < file 

That will print all lines that don't contain any Arabic or Cyrillic characters only. The -CS tells perl that STDIN, STDOUT and STDERR are all unicode. The -ne means "read each input line and apply the script given by -e. The /foo/ || bar means do bar if the line does not match foo. In this case, print if the line does not match any Arabic or Cyrillic characters.

Finally, the \p{} is (from man perluniprops):

   The Perl regular expression "\p{}" and "\P{}" constructs give access to
   most of the Unicode character properties. 

This allows you to match a range of characters such as Arabic or Cyrillic or pretty much anything else you can think of. Therefore, the characters class [\p{foo}] will match any character of the script foo. So, [\p{Arabic}\p{Cyrillic}] will match any characters of the two scripts.

  • I should have probably emphasized elegant in my question, just kidding :) As Cyrillic as it seems, this actually works and enables me to automate the cleaning process. I will accept this answer but some explanation and any additional solution is most welcome. Now I think, i.imgur.com/zAe4U53.png – Gani Simsek Oct 21 '14 at 17:44
  • @GaniSimsek I know, I know :). The user I asked is both a serious Perl guru (he writes the O'reilly books) and has a particular interest in language and computational linguistics. While I know enough Perl do be able to decipher this, I don't know enough to do it just by glancing. In the meantime, I have provided a shorter version that also works with your example and that I am actually capable of explaining. Hope that helps. – terdon Oct 21 '14 at 18:05
  • Thanks, it indeed helps. And what if I wanted to extend unwanted languages to also include Chinese/Japanase/Korean? Trying your version as $ perl -CS -ne '/[\p{Arabic}\p{Cyrillic}\p{CJK}]+/ || print' < file or p{Block: CJK} gives this error: Can't find Unicode property definition "CJK" at -e line 1, <> line 1. I searched for this error but couldn't find a solution. – Gani Simsek Oct 21 '14 at 18:46
  • @GaniSimsek I really don't know. I only just found out about all this when I tried to answer your question. perl -CS -ne '/[\p{Arabic}\p{Cyrillic}\p{CJK}]+/ || print' < file works for me but I don't have any Japanese, Chinese or Korean to test with. Presumably, your perl doesn't have those. Not sure what you would need to install to get them. I suggest you ask on Stack Overflow since this is becoming a pure programming issue. – terdon Oct 21 '14 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.