I'm using fold -w 3 to split a line into multiple 3 chars long, however with the GNU implementation, it does not work for text with multi-byte characters it seems.

How can I achieve the above with sed?

I've come up with sed -r 's/^(.{0,3})(.*)/\1\n\2/g' however this only does a single replacement:

echo "111222333444555666" | sed -r 's/^(.{0,3})(.*)/\1\n\2/g' 
111
222333444555666

Additional examples:

echo "ĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄ" | sed -r 's/^(.{0,3})(.*)/\1\n\2/g' 
ĄĄĄ
ĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄ

And fold with it's corrupting behavior:

echo "ĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄ" | fold -w 3                         
Ą�
�Ą
Ą�
�Ą
Ą�
  • echo "ĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄ" | fold -b6 – Jesse_b Jul 7 at 17:35
  • @Jesse_b that'll only work for strings with all 2-byte characters – phuclv Jul 8 at 1:44
up vote 6 down vote accepted

Short grep approach:

echo "ĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄ" | grep -Eo '.{1,3}'
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄ

To retain only 3-char sequences: ... | grep -Eo '.{3}'

Note that the issue is not with the Unicode character set, but with characters that are encoded on 2 or more bytes (as well as characters that have a width other than one cell).

UTF-8 is an encoding of Unicode where characters U+0080 to U+10FFFFF are encoded on 2 or more bytes. Unicode characters U+0000 to U+007F are the same as in ASCII and in UTF-8 are encoded on a single byte (same as in ASCII) and are not a problem here.

There are other encodings of the Unicode charset (like iso8859-1, single-byte but limited to characters U+0000 to U+00FF, or GB18030, multi-byte), and there are other non-Unicode character sets with multi-byte encodings.

You can tell what character encoding is used in your locale with the locale charmap command.

The GNU implementation of fold currently only works correctly with single-byte characters. The fold of most other systems don't have that problem. Many can even handle characters with zero or double display width.

The busybox implementation of fold has supported UTF-8 (not other multi-byte charmaps though) since 2010.

  • On FreeBSD or Solaris:

    $ echo $'a\u0301bcde' | fold -w3
    ábc
    de
    
  • with busybox fold:

    $ echo $'a\u0301bcde' | busybox fold -w3
    áb
    cde
    
  • with GNU fold:

    $ echo $'a\u0301bcde' | fold -w3
    á
    bcd
    e
    

U+0301 is a combining acute accent. It has a null width and in UTF-8 is encoded on 2 bytes (0xcc 0x81). So, that á ($'a\u0301') is one grapheme cluster of width 1 made of 2 characters encoded on 3 bytes, hence the 3 different behaviours, the most correct of which is FreeBSD/Solaris' here.

Using sed:

$ echo "ĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄ" | sed 's/.../&\n/g'
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄ

Or more general solution(s) (easier to define the number of characters):

sed    's/.\{3\}/&\n/g'             # Using BRE (basic) syntax
sed -E 's/.{3}/&\n/g'               # Using ERE (extended) syntax.

Found a solution:

echo "ĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄ" | sed -r 's/(.{0,3}){1}/&\n/g' 
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
  • 1
    That seems more complicated than it needs to be - you don't seem to be actually using the capture group, and isn't <thing>{1} equivalent to just <thing>? – steeldriver Jul 7 at 17:42
  • (.{0,3}){1} should be .{1,3} – phuclv Jul 8 at 1:45

Just because...

$ echo "ĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄĄ" | gawk '{$1=$1} 1' FPAT=".{,3}" OFS="\n"
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄĄ
ĄĄ

Here is a POSIX solution:

awk '{gsub(/.{5}/, "&\n")} 1'

Interestingly, the Awk solution is more performant than fold.

http://stackoverflow.com/questions/1187078/-/23599010

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.