Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I wrote a regular expression which works well in a certain program (grep, sed, awk, perl, python, ruby, ksh, bash, zsh, find, emacs, vi, vim, gedit, …). But when I use it in a different program (or on a different unix variant), it stops matching. Why?

share|improve this question

1 Answer 1

up vote 37 down vote accepted

Unfortunately, for historical reasons, different tools have slightly different regular expression syntax, and sometimes some implementations have extensions that are not supported by other tools. While there is a common ground, it seems like every tool writer made some different choices.

The consequence is that if you have a regular expression that works in one tool, you may need to modify it to work in another tool. The main differences between common tools are:

  • whether the operators +?|(){} require a backslash;
  • what extensions are supported beyond the basics .[]*^$ and usually +?|()

In this answer, I list the main standards. Check the documentation of the tools you're using for the details.

Wikipedia's comparison of regular expression engines has a table listing the features supported by common implementations.

Basic regular expressions (BRE)

Basic regular expressions are codified by the POSIX standard. It is the syntax used by grep, sed and vi. This syntax provides the following features:

  • ^ and $ match only at the beginning and end of a line.
  • . matches any character (or any character except a newline).
  • […] matches any one character listed inside the brackets (character set). If the first character after the opening bracket is a ^, the characters which are not listed are matched instead.
  • Backslash before any of ^$.*\[ quotes the next character.
  • * matches the preceding character or subexpression 0, 1 or more times.
  • \(…\) is a syntactic group, for use with the * operator or backreferences and \DIGIT replacements.
  • Backreferences \1, \2, … match the exact text matched by the corresponding group, e.g. \(fo*\)\(ba*\)\1 matches foobaafoo but not foobaafo. There is no standard way to refer to the 10th group and beyond (the standard meaning of \10 is the first group followed by a 0).

The following features are also standard, but missing from some restricted implementations:

  • \{m,n\} matches the preceding character or subexpression between m to n times; n or m can be omitted, and \{m\} means exactly m.
  • Inside brackets, character classes can be used, for example [[:alpha:]] matches any letter.

The following are common extensions (especially in GNU tools), but they are not found in all implementations. Check the manual of the tool you're using.

  • \| for alternation: foo\|bar matches foo or bar.
  • \? and \+ match the preceding character or subexpression at most 1 time, or at least 1 time respectively.
  • \n matches a newline, \t matches a tab, etc.
  • \w matches any word constituent and \W matches any character that isn't a word constituent.
  • \< and \> match the empty string only at the beginning or end of a word respectively; \b matches either, and \B matches where \b doesn't.

Note that tools without the \| operator do not have the full power of regular expressions. With backreferences, the power is slightly bigger than regular expressions in the mathematical sense.

Extended regular expressions (ERE)

Extended regular expressions are codified by the POSIX standard. Their major advantage over BRE is regularity: all standard operators are bare punctuation characters, a backslash before a punctuation character always quotes it. It is the syntax used by awk, grep -E or egrep, GNU sed -r, and bash's =~ operator. This syntax provides the following features:

  • ^ and $ match only at the beginning and end of a line.
  • . matches any character (or any character except a newline).
  • […] matches any one character listed inside the brackets (character set). If the first character after the opening bracket is a ^, the characters which are not listed are matched instead. Character classes can be used but are missing from a few implementations.
  • (…) is a syntactic group, for use with * or \DIGIT replacements.
  • | for alternation: foo|bar matches foo or bar.
  • *, + and ? matches the preceding character or subexpression a number of times: 0 or more for *, 1 or more for +, 0 or 1 for ?.
  • Backslash quotes the next character if it is not alphanumeric.
  • {m,m} matches the preceding character or subexpression between m and n times (missing from some implementations); n or m can be omitted, and {m} means exactly m.
  • Some common extensions as in BRE: \DIGIT backreferences (notably absent in awk); special characters \n, \t, etc.; word boundaries \b and \B, word constituents \b and \B, …

PCRE (Perl-compatible regular expressions)

PCRE are extensions of ERE, originally introduced by Perl and adopted by many modern tools and programming languages, usually via the PCRE library. See the Perl documentation for nice formatting with examples. Not all features of the latest version of Perl are supported by PCRE (e.g. Perl code execution is only supported in Perl), see the PCRE manual for a summary of supported features.

Emacs

Emacs's syntax is intermediate between BRE and ERE. In addition to Emacs, it is the default syntax for -regex in GNU find. Emacs offers the following operators:

  • ^, $, ., […], *, +, ? as in ERE
  • \(…\), \|, \{…\}, \DIGIT as in BRE
  • more backslash-letter sequences; \< and \> for word boundaries; and more in recent versions of Emacs, that are often not supported in other engines with an Emacs-like syntax.

Shell globs

Shell globs (wildcards) perform pattern matching with a syntax that is completely different from regular expressions and less powerful. POSIX patterns include the following features:

  • ? matches any single character.
  • […] is a character set as in common regular expression syntaxes. Some shells do not support character classes. Some shells require ! instead of ^ to negate the set.
  • * matches any sequence of characters
  • Backslash quotes the next character.

Ksh offers additional features which give its pattern matching the full power of regular expressions. These features are also available in bash after running shopt -s extglob. Zsh has a different syntax but can also support ksh's syntax after setopt ksh_glob.

share|improve this answer
    
backreferences (\1, \2...) are standard in BRE. –  Stéphane Chazelas May 5 at 6:34
    
Other rich REs you may want to mention are vim's and AT&T libast (as in ksh93) ones. –  Stéphane Chazelas May 5 at 6:35
    
@mikeserv Extremely old versions of ed and grep lacked them, but that was before my time, so I think I can safely list them as something you can count on in BRE. I've edited my answer. –  Gilles Sep 7 at 1:41
    
@StéphaneChazelas Apart from vim, what program uses vim regexps? Apart from ksh, what program uses libast? –  Gilles Sep 7 at 1:42
    
all of the AT&T tool set uses the AT&T REs (grep, tw, expr...). Except for ksh, that toolset is rarely found outside of AT&T though. –  Stéphane Chazelas Sep 7 at 14:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.