4

For a while, I've been dealing with Chinese unicode text. Of course, the usual rules apply. I can grep for characters the same way I'd do so for words. This is very useful to me.

But there's one thing I haven't figured out yet. And I don't know if it's even possible.

It stands to reason that CJK would not be amenable to all kinds of splitting. But line splitting works, of course, using split -l.

What I want to do is be able to split an arbitrary number of characters, though.

My understanding of Chinese unicode is that every glyph is the same number of bytes in size. As such, there should be some magic number of bytes, a least common multiple, which would allow me to use split -b, right?

I used trial and error once, hoping to arrive at that number, but was not able to do so. Instead, the characters themselves were split, such that splitting CJK a file in two.

For instance, given a file called 'dunting' which contains only the string 洞庭湖, using split ends up yielding what is essentially nonsense. One of the characters even becomes 溭 during the split...

3

Each character is three bytes wide, as shown in this xxd output:

$ xxd chinese-bytes
0000000: e6b4 9ee5 baad e6b9 96                   .........

split -b3 works for me.

$ split -b3 chinese-bytes
$ echo xa?
xaa xab xac
$ cat xaa; echo
洞
$ cat xab; echo
庭
$ cat xac; echo
湖
  • Did you try adding a newline in there? – Ignacio Vazquez-Abrams Feb 7 '12 at 0:44
  • @IgnacioVazquez-Abrams With a newline it just creates a fourth file with the newline. I don't see how that relates to the question anyway. – Chris Down Feb 7 '12 at 0:54
  • And then it breaks the next character after the newline. – Ignacio Vazquez-Abrams Feb 7 '12 at 0:56
  • @IgnacioVazquez-Abrams This is not referenced anywhere in the question. In fact, the final paragraph of the question illustrates an example where there is nothing after the newline, and yet there are still claims of it changing to nonsense characters. – Chris Down Feb 7 '12 at 1:02
  • 1
    Although this answers the question in the narrowest sense possible (ipso facto 1 chinese char = 3 bytes), it fundamentally misunderstands the most basic aspects of unicode & text processing in general (not to mention the question), rendering it at best meaningless & at worst dangerous (all reinforced by subsequent comments). Simply test using 1 or 2 consecutive newlines (i.e., 1 or two bytes) between your 3-byte unicode chars, and game over. Or consider that almost all chinese texts contain some latin characters. (eg cat chinese-bytes chinese-bytes > tst2; split -b3 tst2 => garbage) – michael Dec 30 '17 at 12:55
2

As far as I know, all Chinese characters are 3 bytes long when encoded in UTF-8, the normal Unicode encoding on unix. But non-Chinese characters such as spaces and newlines may be a different width (the basic control characters, as well as arabic digit forms and others, are a single byte wide). The split utility only understands fixed numbers of bytes, so it'll cut indiscriminately out of alignment.

You'll need to use a more sophisticated tool to split every 42 characters. Here's a Perl snippet that should do the trick (untested). Note that is treats every character equally: a Chinese character counts for 1, so does a newline.

perl -CDS -e '
    $n = 0;
    while (read STDIN, $buf, 42) {
        open OUT, sprintf("> output-$n.txt") or die;
        print OUT $buf;
        close OUT or die;
        ++$n;
    }'
-1

In the mac terminal I'm using egrep -o '.'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.