In a regular expression, which characters need escaping?

Question

In general, which characters in a regular expression need escaping?

For example, the following is not syntactically correct:

echo '[]' | grep '[]'
grep: Unmatched [ or [^

This, however, is syntatically correct:

echo '[]' | grep '\[]'
[]

Is there any documentation on which characters should be escaped in a regular expression, and which should not?

Answer 1

This depends on the application. In your example [ must be quoted as an argument for grep but not echo.

For the shell (from the POSIX specs):

Quoting is used to remove the special meaning of certain characters or words to the shell. Quoting can be used to preserve the literal meaning of the special characters in the next paragraph, prevent reserved words from being recognized as such, and prevent parameter expansion and command substitution within here-document processing (see Here-Document).

The application shall quote the following characters if they are to represent themselves:

|  &  ;  <  >  (  )  $  `  \  "  '  <space>  <tab>  <newline>

and the following may need to be quoted under certain circumstances. That is, these characters may be special depending on conditions described elsewhere in this volume of IEEE Std 1003.1-2001:

*   ?   [   #   ˜   =   %

The various quoting mechanisms are the escape character, single-quotes, and double-quotes. The here-document represents another form of quoting; see Here-Document.

Specific programs (using regexes, perl, awk) could have additional requirements on escaping.

Answer 2

Each application will have its own set of 'special' characters. The issue that you ran into was with grep not the shell. For which characters need to be quoted in grep, read the manpage's section on "REGULAR EXPRESSIONS".

For the shell, that characters that should be quoted are:

;'"`#$&*?[]<>{}\

and any whitespace.

Depending on the shell, other characters may need to be quoted as well:

!^%

Look under "SHELL GRAMMAR" on the shell's manpage.

Answer 3

grep uses BRE as its regex method. There is good documentation on it here, a general rundown would be "escape any special character or metacharacter to get its literal, escape to create escape sequences (\n, \r, etc)", although this is not always true, for example, you have to escape ( and ) to get their special meaning (backreference).