Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

It seems e.g.
cat sed_data.txt | sed 's/\b[0-9]\{3\}\b/NUMBER/g'
that I must escape characters to form a regular expression. In this case I had to escape braces in order to be interpreted as a number of times.
Why? I was expecting that everything would be a regex character unless escaped. I.e. the opposite.

share|improve this question
    
There was a post about searching in Vim that somewhat covers this question, the short version being "it depends on the command's implementation"... unix.stackexchange.com/questions/90345/… –  Drav Sloan Sep 14 '13 at 18:35
    
@DravSloan:I am not sure it is the same.In Vim you search text by default and you need to escape to search for regex.But in this case the format s/regex//g already expects a regex and I would expect that it is text that would need to be escaped –  Jim Sep 14 '13 at 18:39

2 Answers 2

This is because sed uses POSIX BREs (Basic Regular Expressions) as opposed to the EREs (Extended Regular Expressions) you're probably used to from Perl or friends.

From the sed(1) man page:

REGULAR EXPRESSIONS
       POSIX.2 BREs should be supported, but they aren't completely because of
       performance problems.  The \n sequence in a regular expression  matches
       the newline character, and similarly for \a, \t, and other sequences.

Relevant quote from the above link:

The Basic Regular Expressions or BRE flavor standardizes a flavor similar to the one used by the traditional UNIX grep command. This is pretty much the oldest regular expression flavor still in use today. One thing that sets this flavor apart is that most metacharacters require a backslash to give the metacharacter its flavor. Most other flavors, including POSIX ERE, use a backslash to suppress the meaning of metacharacters.

Quoted verbatim from Craig Sanders' comment:

Note that in GNU sed at least, you can tell sed to use extended regexps with the -r or --regexp-extended command line option. This is useful if you want to avoid uglifying your sed script with excessive escaping.

share|improve this answer
1  
Note that in GNU sed at least, you can tell sed to use extended regexps with the -r or --regexp-extended command line option. This is useful if you want to avoid uglifying your sed script with excessive escaping. –  cas Sep 15 '13 at 0:54
    
@CraigSanders Thanks for this. Added to answer. –  Joseph R. Sep 15 '13 at 14:39
    
@CraigSanders, other sed implementations (when they do support EREs, mostly BSDs) tend to use -E for that instead (which makes a lot more sense since that's the same option as for grep. Why GNU sed chose -r is a mystery to me). –  Stéphane Chazelas Sep 18 '13 at 6:16
    
yeah, a mystery to me too. It would make more sense to use -E. and then add -F, -G, and -P to match GNU grep. IMO gawk would benefit from the same RE args too...or at least, -P. –  cas Sep 18 '13 at 11:56

That's for historical reasons.

Regexp were first introduced in Unix in the ed utility in the early 70s. Though ed was based on qed whose implementation by the the same authors understood more complex regexp, ed only understood ^, $, [...], ., * and \ to escape all of the above.

Now, when the need to have more operators arose, a way had to be found to introduce them without breaking backward compatibility. If a script used to use the s ed command as s/foo() {/foo (var) {/g to replace all the instances of foo() { with foo(var) { and you introduced a ( or { operator, that would break that script.

However no script would do s/foo\(\) {/foo\(var\) {/, because that's the same as s/foo() {/foo(var) {/ and there was no reason to escape ( as that was not a RE operator. So introducing a new \( or \{ operator does not break backward compatibility as it's very unlikely to break an existing script using the older syntax.

So, that's what was done. Later, \(...\) was added initially only for the s ed command to do things like s/foo\(.\)/\1bar/ and later as grep '\(.\)\1' (but sill not things like \(xx\)*).

In UnixV7 (1979, so almost a decade later), a new form of regular expressions were added in the new egrep and awk utilities called extended regular expression (since they are new tools, there's no backward compatibility to be broken). At last, it provided with the functionality available in the Ken Thomson's ancient qed (alternation operator |, grouping (..)*) and added a few operators like + and ? (but didn't have the backref feature of the basic regular expressions).

Later the BSDs added \< and \> (to both BRE and ERE), and SysV added \{ and \} to BREs only.

It's not until much later than { and } were added to ERE, by such breaking backward compatibility. Not everyone added it. For instance, GNU awk until version 4.0.0 (2011) didn't support { unless forced into POSIX conformance mode.

when GNU grep was written in the early 90s, it added all the goodies from both BSD and SysV (like \<, {) and instead of having two separate regexp syntax and engine for BRE and ERE, implemented the same operators in both, only the BRE counterparts of (, ?, {, + have to be preceded with a backslash (to be compatible with other BRE implementations). That's why you can do .\+ in GNU grep (though that's not POSIX or supported by other implementations) and you can do (.)\1 in GNU egrep (though that's not POSIX or supported by many other implementations including GNU awk).

Adding \x operators is not the only way to add more operators in a backward compatible way. For instance, perl used (?...). That's still backward compatible with EREs as (?=...) is not valid in EREs, same for .*?. vim for similar operators did it differently by introducing \@= or .\{-} for instance.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.