coreutils that are utf aware?

Question

When I used cut today, I found out it doesn't treat a UTF-8 character as a character, but 3 characters because it's 3 bytes long.

This seems to be generally true for many tools.

Are there versions of the coreutils that are UTF-8 aware?

My locale output:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Here is when cut doesn't work

echo 哈哈 | cut -c 2-
��哈

The right output should be

哈

if cut -c worked with multibyte characters.

Answer 1

GNU coreutils do understand UTF-8 in general. For example echo 哈哈 | wc -m correctly outputs 3 in a UTF-8 locale (note that the option is -m, not -c which for historical reasons means bytes).

This is a bug in cut. Looking at the source of cut, cut on characters is simply not implemented: the -c option is treated as a synonym of -b.

A workaround is to use awk. GNU awk copes with UTF-8 just fine.

awk '{print substr($0,2,length)}'

Answer 2

This looks like a bug in your build/version of coreutils. I can reproduce this on Ubuntu 10.10 Maverick Meerkat but not on Fedora 15.

[patches@holocene ~]$ cat /etc/fedora-release 
Fedora release 15 (Lovelock)
[patches@holocene ~]$ rpm -q coreutils
coreutils-8.10-2.fc15.x86_64
[patches@holocene ~]$ echo 哈哈 | cut -c 2-
哈
[patches@holocene ~]$ sudo chroot /mnt/maverick
root@holocene:/# grep DISTRIB_DESC /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu 10.10"
root@holocene:/# dpkg-query -s coreutils | grep Version
Version: 8.5-1ubuntu3
root@holocene:/# echo 哈哈 | cut -c 2-
��哈

If you're also using Ubuntu, you can report a bug to the Ubuntu coreutils packagers by running the following command:

apport-bug coreutils

Update: Gilles points out in the comments that this is a bug in the upstream version of coreutils that Fedora has patched. You can find their patch here if you would like to try to patch it yourself to get it working.