Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 2 down vote favorite

I recently started processing Cyrillic text, and it's been really difficult.

I couldn't get my Python scripts to work with it at all. And I tried.

PHP worked well, but I don't know PHP. I just managed to hack a few things together, and I still don't feel comfortable in it. (It may become a bit of a mainstay, though, as it's proven unexpectedly useful.)

Of course, grep is out of the question.

Or is it?

That's what this question is about.

I wanted to do this:

alec@ROOROO:~/$ grep '\w\{4\}' cyrillicstuff

...and came up empty handed.

But is there a way I could have returned all words 4 characters or greater, given that they're all in Cyrillic, using good 'ol grep??

share|improve this question
Most tools will be utf8 aware if you have the env var LANG set to something like en_US.utf8. However grep does not appear to be utf8 aware. An alternative would be perl -CDS -ne '/\w{4,}/ && print' cyrillicstuff. Not exactly short, but you could alias it or write a shell function to wrap it. – Patrick Jun 12 '12 at 2:24
1  
@Patrick: grep, at least the GNU one, is perfectly utf-8 aware. It just has different definition of \w, so you can't use that, but have to use the named classes. – Jan Hudec Jun 12 '12 at 5:48
@JanHudec grep -P '\w{4,}' fails to work, while perl -CDS -ne '/\w{4,}/ && print' works just fine. In fact perl demonstrates the EXACT same behavior as grep when you dont give it the -CDS argument (which enables utf8 in perl). – Patrick Jun 12 '12 at 11:55
@Patrick Jan is right. \w does work (and is locale-aware) with GNU grep 2.6.3, but only with basic and extended regexps, not with -P for some reason. Perl doesn't import locale settings from the environment without -CLS. – Gilles Jun 12 '12 at 22:04
@Gilles wait, Jan said \w doesnt work (and even without -P it didnt work here either). Though it does seem that the [[:alnum:]] solution below works, so ya, it was just the -P test that was failing :-) – Patrick Jun 13 '12 at 18:44

1 Answer

up vote 3 down vote accepted

I believe you need to use the unicode-based character classes instead. The locale-aware class for word characters is [:alnum:] and this is used inside character class, so the command would be

grep '[[:alnum:]]\{4\}' cyrillicstuff

and make sure your locale is set to the encoding the file is actually in. You can check with locale command and look for what value it gives for LC_CTYPE category.

This syntax is supported by all tools that use POSIX basic or extended regular expressions like sed, awk etc. and also by perl and "perl compatible regular expressions" used by python and php. The perl and "perl compatible regular expressions" have one additional syntax \pX and \p{xxx}, where X or xxx is a unicode category name, so \pL is the same as [:alpha:] and \p{Uppercase} should be the same as [:upper:]. All unicode categories should be usable.

Ad python. Python is perfectly unicode aware too. In python 3 it should work out of the box, opening files in locale encoding seems to be default there (but I just looked it up, not tested). However in python 2, you have to specify the encodings there manually. They should be set for stdin, stdout and stderr, but for all other files you have to use the codecs.open function and specify the encoding you get from locale.getpreferredencoding() and you have to initialize locales like in C with locale.setlocale(locale.LC_ALL, '').

share|improve this answer
thanks. as a matter of fact, i was using python 3. =( – ixtmixilix Jun 12 '12 at 13:09
@ixtmixilix: I haven't used it myself. There are two parts to using unicode in your case: opening the file so that it returns correct unicode strings (check known codepoint with repr) and using locale-aware character class in the regexp (hm, python3 doc says \w is by default if the pattern is str (unicode in py3) and not bytes). – Jan Hudec Jun 12 '12 at 14:24
thanks for the tip. i had no idea about repr, and neither did anyone i asked. – ixtmixilix Jun 15 '12 at 23:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.