What characters do I need to escape when using sed in a sh script?

Question

Take the following script:

#!/bin/sh
sed 's/(127\.0\.1\.1)\s/\1/' [some file]

If I try to run this in sh (dash here), it'll fail because of the parentheses, which need to be escaped. But I don't need to escape the backslashes themselves (between the octets, or in the \s or \1). What's the rule here? What about when I need to use {...} or [...]? Is there a list of what I do and don't need to escape?

Answer 1

There are two levels of interpretation here: the shell, and sed.

In the shell, everything between single quotes is interpreted literally, except for single quotes themselves. You can effectively have a single quotes between single quotes by writing '\'' (close single quote, one literal single quote, open single quote).

Sed uses basic regular expressions. In a BRE, the characters $.*[\]^ need to be quoted by preceding them by a backslash, except inside character sets ([…]). Letters, digits and (){}+?| must not be quoted (you can get away with quoting some of these in some implementations). The sequences \(, \), \n, and in some implementations \{, \}, \+, \?, \| and other backslash+alphanumerics have special meanings. You can get away with not quoting $^] in some positions in some implementations.

Furthermore, you need a backslash before / if it is to appear in the regex. You can choose an alternate character as the delimiter by writing e.g. s~/dir~/replacement~ or \~/dir~p; you'll need a backslash before the delimiter if you want to include it in the BRE. If you choose a character that has a special meaning in a BRE and you want to include it literally, you'll need three backslashes; I do not recommend this.

In a nutshell, for sed:

• Write the regex between single quotes.
• Use '\'' to search for a single quote.
• Put a backslash before $.*/[\]^ and only those characters.

In the replacement text, & and \ need to be quoted, as do the delimiter and newlines. \ followed by a digit has a special meaning. \ followed by a letter has a special meaning (special characters) in some implementations, and \ followed by some other character means \c or c depending on the implementation.

If the regex or replacement text comes from a shell variable, remember that

• the regex is a BRE, not a literal string;
• in the regex, a newline needs to be expressed as \n;
• in the replacement text, &, \ and newlines need to be quoted;
• the delimiter needs to be quoted.
• Use double quotes for interpolation: sed -e "s/$BRE/$REPL/"

Answer 2

The problem you're experiencing isn't due to shell interpolating and escapes - it's because you're attempting to use extended regular expression syntax without passing sed the -r or --regexp-extended option.

Change your sed line from

sed 's/(127\.0\.1\.1)\s/\1/' [some file]

to

sed -r 's/(127\.0\.1\.1)\s/\1/' [some file]

and it will work as I believe you intend.

By default sed uses uses basic regular expressions (think grep style), which would require the following syntax:

sed 's/\(127\.0\.1\.1\)[ \t]/\1/' [some file]

Answer 3

Unless you want to interpolate a shell variable into the sed expression, use single quotes for the whole expression because they cause everything between them to be interpreted as-is, including backslashes.

So if you want sed to see s/\(127\.0\.1\.1\)\s/\1/ put single quotes around it and the shell won't touch the parentheses or backslashes in it. If you need to interpolate a shell variable, put only that part in double quotes. E.g.

sed 's/\(127\.0\.1\.1\)/'"$ip"'/'

This will save you the trouble of remembering which shell metacharacters are not escaped by double quotes.