Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Take the following script:

#!/bin/sh
sed 's/(127\.0\.1\.1)\s/\1/' [some file]

If I try to run this in sh (dash here), it'll fail because of the parentheses, which need to be escaped. But I don't need to escape the backslashes themselves (between the octets, or in the \s or \1). What's the rule here? What about when I need to use {...} or [...]? Is there a list of what I do and don't need to escape?

share|improve this question

3 Answers 3

up vote 29 down vote accepted

There are two levels of interpretation here: the shell, and sed.

In the shell, everything between single quotes is interpreted literally, except for single quotes themselves. You can effectively have a single quotes between single quotes by writing '\'' (close single quote, one literal single quote, open single quote).

Sed uses basic regular expressions. In a BRE, the characters $.*[\]^ need to be quoted by preceding them by a backslash, except inside character sets ([…]). Letters, digits and (){}+?| must not be quoted (you can get away with quoting some of these in some implementations). The sequences \(, \), \n, and in some implementations \{, \}, \+, \?, \| and other backslash+alphanumerics have special meanings. You can get away with not quoting $^] in some positions in some implementations.

Furthermore, you need a backslash before / if it is to appear in the regex. You can choose an alternate character as the delimiter by writing e.g. s~/dir~/replacement~ or \~/dir~p; you'll need a backslash before the delimiter if you want to include it in the BRE. If you choose a character that has a special meaning in a BRE and you want to include it literally, you'll need three backslashes; I do not recommend this.

In a nutshell, for sed:

  • Write the regex between single quotes.
  • Use '\'' to search for a single quote.
  • Put a backslash before $.*/[\]^ and only those characters.

In the replacement text, & and \ need to be quoted, as do the delimiter and newlines. \ followed by a digit has a special meaning. \ followed by a letter has a special meaning (special characters) in some implementations, and \ followed by some other character means \c or c depending on the implementation.

If the regex or replacement text comes from a shell variable, remember that

  • the regex is a BRE, not a literal string;
  • in the regex, a newline needs to be expressed as \n;
  • in the replacement text, &, \ and newlines need to be quoted;
  • the delimiter needs to be quoted.
  • Use double quotes for interpolation: sed -e "s/$BRE/$REPL/"
share|improve this answer

The problem you're experiencing isn't due to shell interpolating and escapes - it's because you're attempting to use extended regular expression syntax without passing sed the -r or --regexp-extended option.

Change your sed line from

sed 's/(127\.0\.1\.1)\s/\1/' [some file]

to

sed -r 's/(127\.0\.1\.1)\s/\1/' [some file]

and it will work as I believe you intend.

By default sed uses uses basic regular expressions (think grep style), which would require the following syntax:

sed 's/\(127\.0\.1\.1\)[ \t]/\1/' [some file]
share|improve this answer
    
I had this problem again, and forgot to scroll down to find the solution I upvoted last time. Thanks again. –  isaaclw Apr 4 at 20:17

Unless you want to interpolate a shell variable into the sed expression, use single quotes for the whole expression because they cause everything between them to be interpreted as-is, including backslashes.

So if you want sed to see s/\(127\.0\.1\.1\)\s/\1/ put single quotes around it and the shell won't touch the parentheses or backslashes in it. If you need to interpolate a shell variable, put only that part in double quotes. E.g.

sed 's/\(127\.0\.1\.1\)/'"$ip"'/'

This will save you the trouble of remembering which shell metacharacters are not escaped by double quotes.

share|improve this answer
    
I want sed to see s/(127\.0\.1\.1)/..., but putting that in a shell script as-is doesn't work. What you're saying about the shell not touching the parentheses seems wrong. I've edited my question to elaborate. –  detly Feb 28 '12 at 6:14
3  
The shell isn't touching the parentheses. You need the backslases because sed needs to see them. sed 's/(127\.0\.1\.1)/IP \1/' fails because sed needs to see \( and \) for group syntax, not ( and ). –  Kyle Jones Feb 28 '12 at 6:31
    
facepalm It's not in the man page, but it IS in some online manual I found. Is this normal for regex, because I've never had to use it in regex libraries (in, eg. Python)? –  detly Feb 28 '12 at 6:33
3  
For traditional Unix commands, there are basic regular expressions and extended regular expressions. Details. sed uses basic regular expressions, so the backslashes are needed for group syntax. Perl and Python went beyond even extended regular expressions. While I was poking around I found an extremely informative chart that illustrates what a confusing bramble we conjure up when we glibly say "regular expression." –  Kyle Jones Feb 28 '12 at 7:07
1  
I would also add that the only character that cannot be used inside single quotes is a single quote. –  enzotib Feb 28 '12 at 9:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.