Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

In a huge UTF-8 text file, I want to show all lines that contains Japanese kanjis.
What grep (or other) expression does this?

If I am not mistaken, kanjis are the characters between \u4e00 and \u4dbf.

I don't need to show kanas, but showing them too would not be a big problem.

share|improve this question
add comment

2 Answers 2

up vote 9 down vote accepted

It is impossible (without using a huge table) to tell apart a japanese kanji from a Han ideograph not used in Japanese (eg, a chinese or korean variant).

If you just want to detect any Han ideograph in the basic range (\u4e00 to \u9fff) then they are encoded in 3 bytes, the first byte is always between 0xe4 and 0xe9, the second and third bytes between 0x80 and 0xbf.

There are two difficulties here, first you have to tell grep you want to look after bytes and not characters; then you have to type the 0xe4, 0xe9, 0x80 and 0xbf bytes to put them in the regexp expression.

I discovered the -P switch does both; and the line you want is:

grep -P "[\xe4-\xe9][\x80-\xbf][\x80-\xbf]"

and if you want kana too:

grep -P "[\xe4-\xe9][\x80-\xbf][\x80-\xbf]|\xe3[\x81-\x83][\x80-\xbf]"
share|improve this answer
    
That works great! –  Nicolas Raoul Feb 22 '13 at 10:11
add comment

According to the fifth table here, kanji are the caracters between \u4e00 and \u9fff

My implementation of grep doesn't seem to be able to handle unicode characters (that's GNU grep 2.14 on Archlinux), but we can still use \x. You can find the respective codes here or use a tool like hexedit to get them.

For anything in our range of interest above e9 be a5 returned "Invalid collation character" so this is what I've come up with:

grep "["$'\xe4\xb8\x80'"-"$'\xe9\xbe\xa5'"]" file.txt
share|improve this answer
    
+1 That works too, but Pablo was a bit faster... –  Nicolas Raoul Feb 22 '13 at 10:11
1  
That is equivalent to grep "[一-龥]". the problem is it is locale-dependent, and uses collation rules; that is, only elements for which a sorting order is defined can be used in the range or matched. The -P switch just does binary matching, regardless of the locale. A locale-based approach will use a cultural definition of "character range", a binary-based approach will use an encoding value definition of "range". Particularly for alphabetic scripts the output is quite different. (here for Han ideographics it is roughly equivalent) –  Pablo Saratxaga Feb 22 '13 at 11:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.