Problem: I have text file with approximately 1M lines, each line consists of multiple words Some lines contain Russian or Arabic words (lets call them "bad lines"), I want to remove only these bad lines. The others are "good lines".
- There are non-ascii characters in some of good lines, so just removing everything non-ascii doesn't solve the problem.
- Each line is either completely bad or good, so this actually makes things a little bit easier.
So the solution that came to my mind is to just:
sort file.txt > sorted.txt
Anything starting with Russian or Arabic will be listed at the end. And then I will manually check the line they begin and then do
head -n X sorted.txt > clean.txt
to get rid of them. Is there a more elegant way to do this? What other Unix tools can be used to accomplish this task.
kedi cat кошка القط candy şeker конфеты كاندي çağrı resumé
kedi cat candy şeker çağrı resumé