6

I am working on a text written in Italian and Chinese and I need to extract only Chinese characters using AWK. How can I do this?

I tried:

[The range of Chinese Unicode chars is 4E00 thru 9FFF (344 270 200 thru 351 277 277) so the test should be >"\343" and <"\352" (to avoid picking up any 4 char UTF-8 codes)]:

{
f=0;
for ( i=1; i<=length; i++)
if(substr($0, i, 1)>"\343" &&substr($0, i, 1)<"\352")
f = 1 

print $f
}

But there is an error or more errors. I can't find it / them

2
  • 2
    Can you give example input. Jun 26, 2015 at 10:24
  • Why awk instead of e.g. tr?
    – Costas
    Jun 26, 2015 at 10:32

2 Answers 2

13

Your problem is that by filtering on raw bytes in a UTF-8 character stream, you're eating part of a unicode sequence in a UTF-8 file, resulting in an invalid byte sequence. That can't work. Instead, you need to use a tool that understands UTF-8, and apply a filter on the unicode data, rather than the raw bytes.

Since I don't know which implementation of awk you're using, it's impossible for me to tell whether it supports unicode. However, I know that perl is fully unicode-safe, so the following perl one-liner should work:

perl -CS -p -e 's/[^\s\p{Han}]//g'

The \s is for whitespace, which I'm assuming you'll want to see. The \p{Han} bit tells perl that we want to match characters that are declared in Unicode as being used in the Han script (i.e., chinese characters). I don't know if you need any punctuation characters that aren't included in that range; if you do, you may need to add that as well.

We then negate the range with the ^ at the start, and finally encode it in a global substitute command (s///g) where we tell perl to replace instances of the part after the first slash (our negated range, or, "everything not in this range") with the part after the second and before the third (i.e., nothing).

If you don't need to include several ranges, you can drop the [^] construction, and switch to using \P rather than \p, which does the same match inversion.

What's left is the character ranges we entered -- unicode characters in the Han script, plus whitespace.

For more information, see perldoc perlre for an explanation on how perl deals with regular expressions, and perldoc perluniprops for a list of possible unicode properties (the bits you can place inside a \p{} or \P{} construct).

5
  • Surely that doesn't work (it doesn't work on my chinese text sample). Aren't you searching for literal hexadecimal values? when the text in question is in UTF-8 format.
    – Peter.O
    Jun 26, 2015 at 13:16
  • It turns out you need to use -C1 to tell perl to enable unicode on stdin. Missed that, sorry. Edited to fix. Jun 26, 2015 at 14:14
  • 1
    Since you're recommending perl, wouldn't you think perl -CAS -pe 's/\P{Script: Han}//g to be better (it iincludes more than 75000 characters)?
    – ninjalj
    Jun 27, 2015 at 11:56
  • 1
    The Han script includes the blocks: 2E80..2E99 2E9B..2EF3 2F00..2FD5 3005 3007 3021..3029 3038..303A 303B 3400..4DB5 4E00..9FCB F900..FA2D FA30..FA6D FA70..FAD9 20000..2A6D6 2A700..2B734 2F800..2FA1D, so I really would recommend using scripts instead of blocks.
    – ninjalj
    Jun 27, 2015 at 12:04
  • @ninjalj yes, indeed. Edited to reflect that. Thanks. Jun 30, 2015 at 13:20
1

Using awk you can do it as:

awk '{for(i=1; i<=length;i++) if(substr($0,i,1)>="\xS_INDEX" && substr($0,i,1)<="\xE_INDEX"){printf substr($0,i,1);f=1;} if(f)printf "\n"; f=0}' filename

Here S_INDEX and E_INDEX are the starting and ending index of ascii in hex.

For input:

1243
135
dgfsdaa
125
sdg124
sdf34

Selecting only digits: S_INDEX = 30 and E_INDEX = 39

Output:

1243
135
125
124
34
1
  • 1
    This only works in the 8-bit range. It doesn't help to filter Chinese characters. Jun 26, 2015 at 22:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .