Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

When I used cut today, I found out it doesn't treat a utf-8 character as a character, but 3 characters because it's 3 bytes long.

This seems to be generally true for many tools.

Are there versions of the coreutils that are utf-8 aware?

my locale output

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Here is when cut doesn't work

echo 哈哈 | cut -c 2-
��哈

The right output should be

if cut -c does work on multibyte characters.

share|improve this question
    
Did you set your locale properly? What's the reading of locale (w/o arguments) command invocation? –  alex Jul 2 '11 at 5:00
    
I have updated with the locale. –  Chao Xu Jul 2 '11 at 5:13
    
OK, can you also add some example of your cut command line? –  alex Jul 2 '11 at 5:40
    
Can confirm this on Ubuntu 10.04 with echo ßßßß | cut -c 2- -> �ßßß (LANG=en_US.UTF-8) –  maxschlepzig Jul 2 '11 at 8:52
    
The sad thing is that, 3 years later, this is still true in Ubuntu 13.10... –  Dr. Mike Mar 22 at 10:54

2 Answers 2

up vote 9 down vote accepted

GNU coreutils do understand UTF-8 in general. For example echo 哈哈 | wc -m correctly outputs 3 in a UTF-8 locale (note that the option is -m, not -c which for historical reasons means bytes).

This is a bug in cut. Looking at the source of cut, cut on characters is simply not implemented: the -c option is treated as a synonym of -b.

A workaround is to use awk. GNU awk copes with UTF-8 just fine.

awk '{print substr($0,2,length)}'
share|improve this answer
5  
It would be worth filing a bugreport (even if it ends up being a duplicate) to annoy the coreutils maintainers into actually fixing bugs like this, rather than just adding stupid hacks and toy utilities that nobody needs... –  R.. Jul 3 '11 at 16:27

This looks like a bug in your build/version of coreutils. I can reproduce this on Ubuntu 10.10 Maverick Meerkat but not on Fedora 15.

[patches@holocene ~]$ cat /etc/fedora-release 
Fedora release 15 (Lovelock)
[patches@holocene ~]$ rpm -q coreutils
coreutils-8.10-2.fc15.x86_64
[patches@holocene ~]$ echo 哈哈 | cut -c 2-
哈
[patches@holocene ~]$ sudo chroot /mnt/maverick
root@holocene:/# grep DISTRIB_DESC /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu 10.10"
root@holocene:/# dpkg-query -s coreutils | grep Version
Version: 8.5-1ubuntu3
root@holocene:/# echo 哈哈 | cut -c 2-
��哈

If you're also using Ubuntu, you can report a bug to the Ubuntu coreutils packagers by running the following command:

apport-bug coreutils

Update: Gilles points out in the comments that this is a bug in the upstream version of coreutils that Fedora has patched. You can find their patch here if you would like to try to patch it yourself to get it working.

share|improve this answer
    
I just looked at the source, and it's an upstream bug that Fedora apparently patched. The upstream source simply makes -c an alias of -b. –  Gilles Jul 2 '11 at 10:44
    
@Gilles: Interesting. I linked to Fedora's patch in my answer in case someone wants to try and fix it for themselves. –  Patches Jul 2 '11 at 11:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.