4

Can fold be set to recognize characters instead of bytes? Traditional Chinese characters appear to be encoded in three bytes each (in UTF-8 at least), which means that if fold's -w is not a multiple of three, then the following will occur:

$ cat in.txt
【財經中心、政治中心╱台北報導】看不慣政府施政效率緩慢,鴻海集團董事長郭台銘動念選總統!《壹週刊》報導,在川普勝選當晚,郭召集鴻海高層幹部,進行美國總統大選換人後的應變策略演練,讓人驚訝的是,郭詢問在場幹

$ cat in.txt | fold # -w is 80 by default
【財經中心、政治中心╱台北報導】看不慣政府施政效率緩��
�,鴻海集團董事長郭台銘動念選總統!《壹週刊》報導,在�
��普勝選當晚,郭召集鴻海高層幹部,進行美國總統大選換人
後的應變策略演練,讓人驚訝的是,郭詢問在場幹

fold's default output is a width of 80 columns, and this results in 26 2/3 characters (26 * 3 + 2, or 80 bytes) being printed on each line. Therefore, -w must be set to a multiple of three in order to avoid character breakage. So, at least for fold, columns=bytes. Again, my question is, can fold can be set to honor multi-byte characters? The man page doesn't mention anything about this.

4
  • UTF-8 doesn't have a consistent number of bytes for each character, it uses the least number of bytes necessary to express the codepoint. for example UTF-8 only uses one byte to represent ASCII characters. In UTF-8 the leading byte conveys how many bytes will follow for the character. Nov 17 '16 at 8:08
  • So this is where big-endian and little-endian seem like they would have relevance. ... Guess that's a new research topic for tomorrow. Thanks @the_velour_fog Nov 17 '16 at 8:12
  • @RandyJosleyn This has nothing to do with endianness. Nov 17 '16 at 23:31
  • @Gilles good to know, then. I made that assumption having read something about beginning and ending character bytes a while ago. A little knowledge is a dangerous thing, I guess... Nov 17 '16 at 23:54
4

GNU fold and GNU fmt only understand bytes, not characters. To wrap to a certain number of characters, you can use sed.

sed 's/.\{20\}/&\n/g' <in.txt
【財經中心、政治中心╱台北報導】看不慣政
府施政效率緩慢,鴻海集團董事長郭台銘動念
選總統!《壹週刊》報導,在川普勝選當晚,
郭召集鴻海高層幹部,進行美國總統大選換人
後的應變策略演練,讓人驚訝的是,郭詢問在
場幹

If you wanted to break at whitespace (useful for many languages), here's a quick-and-dirty awk script.

awk '
    BEGIN {width = 20}
    NF == 0 {column = 0; print}
    {
        split($0, a);
        for (i in a) {
            w = length(a[i]) + 1;
            column += w;
            if (column > width) {column = w; print ""};
            if (column != w) printf " ";
            printf "%s", a[i];
        }
    }
    END {if (column) print ""}'

In any case make sure that your locale settings are correct. Specifically, LC_CTYPE must designate the right character encoding, e.g. LC_CTYPE=en_US.utf8 or LC_CTYPE=zh_CN.utf8 (any language code that's available on your system will do) for Unicode encoded as UTF-8.

Note that this counts characters, not screen width. Even fixed-width fonts can have double-width characters and this is typically done for Chinese characters, so e.g. a character width of 20 for the text above occupies 40 columns on typical terminals.

3

If you have vim on your system (which can handle UTF-8 characters) you could try this:

insert these vim ex mode commands into a file called fold.vim

set formatoptions+=m
" uncomment and update the line below to set line width to e.g. 60 chars
" textwidth=60
normal! gqG<Esc>
wq

then if you have your chinese characters in a file called chinese_chars then you can run this from your prompt.

vim -e chinese_chars < fold.vim

where vim -e means start vim in "ex" mode and read in the ex commands from STDIN

vimscript explanation

from :h fo-table

m Also break at a multi-byte character above 255. This is useful for Asian text where every character is a word on its own.

So what does that mean?

by default the gq command won't work on characters whose decimal number is above 255. e.g. this char: has a decimal value of 22,577.
Running set formatoptions+=m means breaking will work with the text in chinese_chars.

refer:
http://vim.1045645.n5.nabble.com/textwidth-doesn-t-work-for-Chinese-td1142655.html

For me this edits the file chinese_chars inplace and breaks lines at 79 or 80 chars - the default for gq normal mode command.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.