24

I am on Windows, but I guess my question is still rightly placed here.

C:\Users\User>grep --version
GNU grep 2.6.3

C:\Users\User>sed --version
GNU sed version 4.2.1

I noticed that the following works (outputting here):

echo here | grep -E "\w+"
echo here | grep -E "[her]+"

But, this does not work (outputting nothing):

echo here | grep -E "[\w]+"

This again does (outputting here):

echo here | grep -P "[\w]+"

So [\w] is something specific to Perl regular expressions, I assume. Is that correct?

So, let's talk sed. This works (outputting gone):

echo here | sed -r "s/\w+/gone/"
echo here | sed -r "s/[her]+/gone/"

And again, this does not (outputting here):

echo here | sed -r "s/[\w]+/gone/"

Now, how can I activate Perl regular expressions for sed -- is there any way?

11

Different tools and versions thereof support different variants of regular expressions. The documentation of each will tell you what they support.

Standards exist so that one can rely on a minimum set of features that are available across all conforming applications.

For instance, all modern implementations of sed and grep implement basic regular expressions as specified by POSIX (at least one version or the other of the standard, but that standard has not evolved a lot in that regard in the last few decades).

In POSIX BRE and ERE, you have the [:alnum:] character class. That matches letters and digits in your locale (note that often includes a lot more than a-zA-Z0-9 unless the locale is C).

So:

grep -x '[[:alnum:]_]\{1,\}'

matches one or more alnums or _.

[\w] is required by POSIX to match either backslash or w. So you won't find a grep or sed implementation where that's available (unless via non-standard options).

The behaviour for \w alone is not specified by POSIX, so implementations are allowed to do what they want. GNU grep added that a long time ago.

GNU grep used to have its own regexp engine however it now uses the GNU libc's one (though it does embed its own copy).

It's meant to match alnums and underscore in your locale. However, it currently has a bug in that it only matches single-byte characters (for instance, not é in a UTF-8 locale even though that's clearly a letter and even though it does match é in all the locales where é is a single character).

There also is a \w regexp operator in perl regexp and in PCRE. PCRE/perl are not POSIX regular expressions, they're just another thing altogether.

Now, with the way GNU grep -P uses PCRE, it's got the same issue as without -P. It can be worked around there though by using (*UCP) (though that also has side-effects in non-UTF8 locales).

GNU sed also uses the GNU libc's regexs for its own regexps. It uses it in such a way though that it doesn't have the same bug as GNU grep.

GNU sed doesn't support PCREs. There's some evidence in the code that it has been attempted before, but it doesn't seem to be on the agenda anymore.

If you want Perl's regular expressions, just use perl though.

Otherwise, I'd say that rather than trying to rely on a bogus non-standard feature of your particular implementation of sed/grep, it would be better to stick with the standard and use [_[:alnum:]].

  • [_[:alnum:]] is a nice workaround which allows me to extend it just like [\w/] ([_[:alnum:]/] in that case). – bers May 8 '15 at 14:57
  • 1
    This answer is now outdated with regards to the limitations of GNU grep. – Stéphane Chazelas May 18 '17 at 18:43
7

You are correct - \w is part of PCRE - perl compatible regular expressions. It's not part of the 'standard' regex though. http://www.regular-expressions.info/posix.html

Some versions of sed may support it, but I'd suggest the easiest way is to just use perl in sed mode by specifying the -p flag. (Along with the -e). (More detail in perlrun)

But you don't need [] around it in that example - that's for groups of valid stuff.

echo here  | perl -pe 's/\w+/gone/'

Or on Windows:

C:\>echo here  | perl -pe "s/\w+/gone/"
gone
C:\>echo here  | perl -pe "s/[\w\/]+/gone/"
gone

See perlre for more PCRE stuff.

You can get perl here: http://www.activestate.com/activeperl/downloads

  • Please note the difference between \w and [\w] in my question. I will update it with outputs of each command to make clear which one is working and which one is not. In particular, sed understands \w, but not [\w]. Also, I need [\w] to work because I want to use [\w/] for example. – bers May 8 '15 at 14:47
  • In which case, it's probably a quoting problem. Either way - perl can do it :). – Sobrique May 8 '15 at 14:48
  • Thanks! Stéphane Chazelas' answer is a little closer to what I asked for (since I don't have perl installed - a du*b Windows user, I guess), so I accepted his answer. – bers May 8 '15 at 14:59
  • That's ok - but I'd recommend getting Perl installed on Windows. It's one of the first things that goes on mine, and I find it extremely helpful. – Sobrique May 8 '15 at 15:08
  • \w was in GNU grep (in the 80s) before being in perl and in GNU emacs probably even before that. – Stéphane Chazelas May 8 '15 at 15:35
1

I suspect that grep and sed are deciding differently when to apply the [] and when to expand the \w. In perl regex \w means any word character, and [] define a group to apply any of the characters within as a match. If you "expand" the \w before the [] it will be a character class of all the word characters. If, instead you do [] first you will have a character class with two characters \ and w so it would match any pattern containing one or more of those two characters.

So it seems that sed is seeing the [] and treating it as containing the exact chars to match instead of honoring the special sequence \w as perl and grep do. Of course, the [] are completely unnecessary in this example, but one could perhaps imagine cases where it would be important, but then you could make it work with parens and ors.

  • I would be surprised if that were so. \ is an escape code, and you'd use it for escaping delimiters. Inherently that means it has to have a higher precedence than any thing else. I think it more likely that it's not implemented because \w isn't part of the regular expression spec – Sobrique May 8 '15 at 14:40
  • Well, empirically it seems to be the case using gnu sed for me: echo whe\\ere | sed -r 's/[\w]+/gone/g gives me gonehegoneere as though it is matching each of the ` and w` and doing the substitution – Eric Renouf May 8 '15 at 14:45
  • I can confirm what Eric Renouf is seeing. So we want to unescape the backslash somehow? :) – bers May 8 '15 at 14:51
  • I don't think that's the right answer. Sed just doesn't support mixing the different types of character class definitions, so the answer is if you must use both types of character classes pick another tool, or if you're picking sed use the syntax it supports – Eric Renouf May 8 '15 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.