Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

For a while, I've been dealing with Chinese unicode text. Of course, the usual rules apply. I can grep for characters the same way I'd do so for words. This is very useful to me.

But there's one thing I haven't figured out yet. And I don't know if it's even possible.

It stands to reason that CJK would not be amenable to all kinds of splitting. But line splitting works, of course, using split -l.

What I want to do is be able to split an arbitrary number of characters, though.

My understanding of Chinese unicode is that every glyph is the same number of bytes in size. As such, there should be some magic number of bytes, a least common multiple, which would allow me to use split -b, right?

I used trial and error once, hoping to arrive at that number, but was not able to do so. Instead, the characters themselves were split, such that splitting CJK a file in two.

For instance, given a file called 'dunting' which contains only the string 洞庭湖, using split ends up yielding what is essentially nonsense. One of the characters even becomes 溭 during the split...

share|improve this question
add comment

3 Answers 3

Each character is three bytes wide, as shown in this xxd output:

$ xxd chinese-bytes
0000000: e6b4 9ee5 baad e6b9 96                   .........

split -b3 works for me.

$ split -b3 chinese-bytes
$ echo xa?
xaa xab xac
$ cat xaa; echo
洞
$ cat xab; echo
庭
$ cat xac; echo
湖
share|improve this answer
    
Did you try adding a newline in there? –  Ignacio Vazquez-Abrams Feb 7 '12 at 0:44
    
@IgnacioVazquez-Abrams With a newline it just creates a fourth file with the newline. I don't see how that relates to the question anyway. –  Chris Down Feb 7 '12 at 0:54
    
And then it breaks the next character after the newline. –  Ignacio Vazquez-Abrams Feb 7 '12 at 0:56
    
@IgnacioVazquez-Abrams This is not referenced anywhere in the question. In fact, the final paragraph of the question illustrates an example where there is nothing after the newline, and yet there are still claims of it changing to nonsense characters. –  Chris Down Feb 7 '12 at 1:02
    
"But line splitting works, of course, using split -l." –  Ignacio Vazquez-Abrams Feb 7 '12 at 1:03
show 2 more comments

As far as I know, all Chinese characters are 3 bytes long when encoded in UTF-8, the normal Unicode encoding on unix. But non-Chinese characters such as spaces and newlines may be a different width (the basic control characters, as well as arabic digit forms and others, are a single byte wide). The split utility only understands fixed numbers of bytes, so it'll cut indiscriminately out of alignment.

You'll need to use a more sophisticated tool to split every 42 characters. Here's a Perl snippet that should do the trick (untested). Note that is treats every character equally: a Chinese character counts for 1, so does a newline.

perl -CDS -e '
    $n = 0;
    while (read STDIN, $buf, 42) {
        open OUT, sprintf("> output-$n.txt") or die;
        print OUT $buf;
        close OUT or die;
        ++$n;
    }'
share|improve this answer
add comment

In the mac terminal I'm using egrep -o '.'

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.