Why do I need to escape regex characters in sed to be interpreted as regex characters?

Question

It seems e.g.
cat sed_data.txt | sed 's/\b[0-9]\{3\}\b/NUMBER/g'
that I must escape characters to form a regular expression. In this case I had to escape braces in order to be interpreted as a number of times.
Why? I was expecting that everything would be a regex character unless escaped. I.e. the opposite.

Answer 1

This is because sed uses POSIX BREs (Basic Regular Expressions) as opposed to the EREs (Extended Regular Expressions) you're probably used to from Perl or friends.

From the sed(1) man page:

REGULAR EXPRESSIONS
       POSIX.2 BREs should be supported, but they aren't completely because of
       performance problems.  The \n sequence in a regular expression  matches
       the newline character, and similarly for \a, \t, and other sequences.

Relevant quote from the above link:

The Basic Regular Expressions or BRE flavor standardizes a flavor similar to the one used by the traditional UNIX grep command. This is pretty much the oldest regular expression flavor still in use today. One thing that sets this flavor apart is that most metacharacters require a backslash to give the metacharacter its flavor. Most other flavors, including POSIX ERE, use a backslash to suppress the meaning of metacharacters.

Quoted verbatim from Craig Sanders' comment:

Note that in GNU sed at least, you can tell sed to use extended regexps with the -r or --regexp-extended command line option. This is useful if you want to avoid uglifying your sed script with excessive escaping.

Answer 2

That's for historical reasons.

Regexp were first introduced in Unix in the ed utility in the early 70s. Though ed was based on qed whose implementation by the the same authors understood more complex regexp, ed only understood ^, $, [...], ., * and \ to escape all of the above.

Now, when the need to have more operators arose, a way had to be found to introduce them without breaking backward compatibility. If a script used to use the s ed command as s/foo() {/foo (var) {/g to replace all the instances of foo() { with foo(var) { and you introduced a ( or { operator, that would break that script.

However no script would do s/foo\(\) {/foo\(var\) {/, because that's the same as s/foo() {/foo(var) {/ and there was no reason to escape ( as that was not a RE operator. So introducing a new \( or \{ operator does not break backward compatibility as it's very unlikely to break an existing script using the older syntax.

So, that's what was done. Later, \(...\) was added initially only for the s ed command to do things like s/foo\(.\)/\1bar/ and later as grep '\(.\)\1' (but sill not things like \(xx\)*).

In UnixV7 (1979, so almost a decade later), a new form of regular expressions were added in the new egrep and awk utilities called extended regular expression (since they are new tools, there's no backward compatibility to be broken). At last, it provided with the functionality available in the Ken Thompson's ancient qed (alternation operator |, grouping (..)*) and added a few operators like + and ? (but didn't have the backref feature of the basic regular expressions).

Later the BSDs added \< and \> (to both BRE and ERE), and SysV added \{ and \} to BREs only.

It's not until much later than { and } were added to ERE, by such breaking backward compatibility. Not everyone added it. For instance, GNU awk until version 4.0.0 (2011) didn't support { unless forced into POSIX conformance mode.

when GNU grep was written in the early 90s, it added all the goodies from both BSD and SysV (like \<, {) and instead of having two separate regexp syntax and engine for BRE and ERE, implemented the same operators in both, only the BRE counterparts of (, ?, {, + have to be preceded with a backslash (to be compatible with other BRE implementations). That's why you can do .\+ in GNU grep (though that's not POSIX or supported by other implementations) and you can do (.)\1 in GNU egrep (though that's not POSIX or supported by many other implementations including GNU awk).

Adding \x operators is not the only way to add more operators in a backward compatible way. For instance, perl used (?...). That's still backward compatible with EREs as (?=...) is not valid in EREs, same for .*?. vim for similar operators did it differently by introducing \@= or .\{-} for instance.