How to use [\w]+ in regular expression in sed?

Question

I am on Windows, but I guess my question is still rightly placed here.

C:\Users\User>grep --version
GNU grep 2.6.3

C:\Users\User>sed --version
GNU sed version 4.2.1

I noticed that the following works (outputting here):

echo here | grep -E "\w+"
echo here | grep -E "[her]+"

But, this does not work (outputting nothing):

echo here | grep -E "[\w]+"

This again does (outputting here):

echo here | grep -P "[\w]+"

So [\w] is something specific to Perl regular expressions, I assume. Is that correct?

So, let's talk sed. This works (outputting gone):

echo here | sed -r "s/\w+/gone/"
echo here | sed -r "s/[her]+/gone/"

And again, this does not (outputting here):

echo here | sed -r "s/[\w]+/gone/"

Now, how can I activate Perl regular expressions for sed -- is there any way?

Answer 1

Different tools and versions thereof support different variants of regular expressions. The documentation of each will tell you what they support.

Standards exist so that one can rely on a minimum set of features that are available across all conforming applications.

For instance, all modern implementations of sed and grep implement basic regular expressions as specified by POSIX (at least one version or the other of the standard, but that standard has not evolved a lot in that regard in the last few decades).

In POSIX BRE and ERE, you have the [:alnum:] character class. That matches letters and digits in your locale (note that often includes a lot more than a-zA-Z0-9 unless the locale is C).

So:

grep -x '[[:alnum:]_]\{1,\}'

matches one or more alnums or _.

[\w] is required by POSIX to match either backslash or w. So you won't find a grep or sed implementation where that's available (unless via non-standard options).

The behaviour for \w alone is not specified by POSIX, so implementations are allowed to do what they want. GNU grep added that a long time ago.

GNU grep used to have its own regexp engine however it now uses the GNU libc's one (though it does embed its own copy).

It's meant to match alnums and underscore in your locale. However, it currently has a bug in that it only matches single-byte characters (for instance, not é in a UTF-8 locale even though that's clearly a letter and even though it does match é in all the locales where é is a single character).

There also is a \w regexp operator in perl regexp and in PCRE. PCRE/perl are not POSIX regular expressions, they're just another thing altogether.

Now, with the way GNU grep -P uses PCRE, it's got the same issue as without -P. It can be worked around there though by using (*UCP) (though that also has side-effects in non-UTF8 locales).

GNU sed also uses the GNU libc's regexs for its own regexps. It uses it in such a way though that it doesn't have the same bug as GNU grep.

GNU sed doesn't support PCREs. There's some evidence in the code that it has been attempted before, but it doesn't seem to be on the agenda anymore.

If you want Perl's regular expressions, just use perl though.

Otherwise, I'd say that rather than trying to rely on a bogus non-standard feature of your particular implementation of sed/grep, it would be better to stick with the standard and use [_[:alnum:]].

Answer 2

You are correct - \w is part of PCRE - perl compatible regular expressions. It's not part of the 'standard' regex though. http://www.regular-expressions.info/posix.html

Some versions of sed may support it, but I'd suggest the easiest way is to just use perl in sed mode by specifying the -p flag. (Along with the -e). (More detail in perlrun)

But you don't need [] around it in that example - that's for groups of valid stuff.

echo here  | perl -pe 's/\w+/gone/'

Or on Windows:

C:\>echo here  | perl -pe "s/\w+/gone/"
gone
C:\>echo here  | perl -pe "s/[\w\/]+/gone/"
gone

See perlre for more PCRE stuff.

You can get perl here: http://www.activestate.com/activeperl/downloads

Answer 3

I suspect that grep and sed are deciding differently when to apply the [] and when to expand the \w. In perl regex \w means any word character, and [] define a group to apply any of the characters within as a match. If you "expand" the \w before the [] it will be a character class of all the word characters. If, instead you do [] first you will have a character class with two characters \ and w so it would match any pattern containing one or more of those two characters.

So it seems that sed is seeing the [] and treating it as containing the exact chars to match instead of honoring the special sequence \w as perl and grep do. Of course, the [] are completely unnecessary in this example, but one could perhaps imagine cases where it would be important, but then you could make it work with parens and ors.