5

Can fold be set to recognize characters instead of bytes? Traditional Chinese characters appear to be encoded in three bytes each (in UTF-8 at least), which means that if fold's -w is not a multiple of three, then the following will occur:

$ cat in.txt
【財經中心、政治中心╱台北報導】看不慣政府施政效率緩慢,鴻海集團董事長郭台銘動念選總統!《壹週刊》報導,在川普勝選當晚,郭召集鴻海高層幹部,進行美國總統大選換人後的應變策略演練,讓人驚訝的是,郭詢問在場幹

$ cat in.txt | fold # -w is 80 by default
【財經中心、政治中心╱台北報導】看不慣政府施政效率緩��
�,鴻海集團董事長郭台銘動念選總統!《壹週刊》報導,在�
��普勝選當晚,郭召集鴻海高層幹部,進行美國總統大選換人
後的應變策略演練,讓人驚訝的是,郭詢問在場幹

fold's default output is a width of 80 columns, and this results in 26 2/3 characters (26 * 3 + 2, or 80 bytes) being printed on each line. Therefore, -w must be set to a multiple of three in order to avoid character breakage. So, at least for fold, columns=bytes. Again, my question is, can fold can be set to honor multi-byte characters? The man page doesn't mention anything about this.

4
  • 1
    UTF-8 doesn't have a consistent number of bytes for each character, it uses the least number of bytes necessary to express the codepoint. for example UTF-8 only uses one byte to represent ASCII characters. In UTF-8 the leading byte conveys how many bytes will follow for the character. Nov 17, 2016 at 8:08
  • So this is where big-endian and little-endian seem like they would have relevance. ... Guess that's a new research topic for tomorrow. Thanks @the_velour_fog
    – kGdmioT
    Nov 17, 2016 at 8:12
  • 1
    @RandyJosleyn This has nothing to do with endianness. Nov 17, 2016 at 23:31
  • @Gilles good to know, then. I made that assumption having read something about beginning and ending character bytes a while ago. A little knowledge is a dangerous thing, I guess...
    – kGdmioT
    Nov 17, 2016 at 23:54

2 Answers 2

5

GNU fold and GNU fmt only understand bytes, not characters. To wrap to a certain number of characters, you can use sed.

sed 's/.\{20\}/&\n/g' <in.txt
【財經中心、政治中心╱台北報導】看不慣政
府施政效率緩慢,鴻海集團董事長郭台銘動念
選總統!《壹週刊》報導,在川普勝選當晚,
郭召集鴻海高層幹部,進行美國總統大選換人
後的應變策略演練,讓人驚訝的是,郭詢問在
場幹

If you wanted to break at whitespace (useful for many languages), here's a quick-and-dirty awk script.

awk '
    BEGIN {width = 20}
    NF == 0 {column = 0; print}
    {
        split($0, a);
        for (i in a) {
            w = length(a[i]) + 1;
            column += w;
            if (column > width) {column = w; print ""};
            if (column != w) printf " ";
            printf "%s", a[i];
        }
    }
    END {if (column) print ""}'

In any case make sure that your locale settings are correct. Specifically, LC_CTYPE must designate the right character encoding, e.g. LC_CTYPE=en_US.utf8 or LC_CTYPE=zh_CN.utf8 (any language code that's available on your system will do) for Unicode encoded as UTF-8.

Note that this counts characters, not screen width. Even fixed-width fonts can have double-width characters and this is typically done for Chinese characters, so e.g. a character width of 20 for the text above occupies 40 columns on typical terminals.

3

If you have vim on your system (which can handle UTF-8 characters) you could try this:

insert these vim ex mode commands into a file called fold.vim

set formatoptions+=m
" uncomment and update the line below to set line width to e.g. 60 chars
" textwidth=60
normal! gqG<Esc>
wq

then if you have your chinese characters in a file called chinese_chars then you can run this from your prompt.

vim -e chinese_chars < fold.vim

where vim -e means start vim in "ex" mode and read in the ex commands from STDIN

vimscript explanation

from :h fo-table

m Also break at a multi-byte character above 255. This is useful for Asian text where every character is a word on its own.

So what does that mean?

by default the gq command won't work on characters whose decimal number is above 255. e.g. this char: has a decimal value of 22,577.
Running set formatoptions+=m means breaking will work with the text in chinese_chars.

refer:
http://vim.1045645.n5.nabble.com/textwidth-doesn-t-work-for-Chinese-td1142655.html

For me this edits the file chinese_chars inplace and breaks lines at 79 or 80 chars - the default for gq normal mode command.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .