Bash sed replace double dollar sign $$ extended regular expressions

Question

This command:

echo '$$foo=bar' | sed -E "s/(\$\$foo=).*/\1$(echo hello)/"

outputs:

$$foo=bar

If I change the sed's quotes to single and enclose the command in double quotes:

echo '$$foo=bar' | sed -E 's/(\$\$foo=).*/\1'"$(echo hello)"'/'

it outputs the desired result:

$$foo=hello

So I guess the problem lies in the extended regular expressions and quotes, unless I'm missing something obvious.

1. Is concatenating single and double quotes in sed command substitution good practice (sed 's/foo/'"$(command)"'/')?

2. How can I escape double dollar signs in sed with extended regular expressions?

Answer 1

POSIX has this to say about the difference in how $ is interpreted in basic and extended regular expressions:

Basic regular expressions (BREs):

A <dollar-sign> ( $ ) shall be an anchor when used as the last character of an entire BRE. The implementation may treat a <dollar-sign> as an anchor when used as the last character of a subexpression. The <dollar-sign> shall anchor the expression (or optionally subexpression) to the end of the string being matched; the <dollar-sign> can be said to match the end-of-string following the last character.

Extended regular expressions (EREs):

A <dollar-sign> ( $ ) outside a bracket expression shall anchor the expression or subexpression it ends to the end of a string; such an expression or subexpression can match only a sequence ending at the last character of a string. For example, the EREs ef$ and (ef$) match ef in the string abcdef, but fail to match in the string cdefab, and the ERE e$f is valid, but can never match because the f prevents the expression e$ from matching ending at the last character.

Conclusion: In a BRE, the $ character matches itself unless it's the last character of the expression or sub-expression (in which case it anchors the (sub-)expression to the end of the line). In an ERE, the $ character always anchors to the end of the line.

When you use

sed -E "s/(\$\$foo=).*/\1$(echo hello)/"

your ERE (since you use -E) is ($$foo=).* and this expression will never match (the POSIX text above contains the example e$f which is similar).

Your command

sed "s/\$\$foo/\$\$hello/"

uses the BRE $$foo which will match the literal string $$foo since the $ characters are not at the end of the expression.

To match a single $ character in an extended regular expression, use \$ or [$]. To escape that for the shell in a double quoted string, use \\\$ (an escaped backslash followed by an escaped dollar sign) or [\$], i.e.,

sed -E "s/(\\\$\\\$foo=).*/\1$(echo hello)/"

or

sed -E "s/([\$][\$]foo=).*/\1$(echo hello)/"

(The backslash in \1 does not need escaping since backslashes only act as an escape character in a double quoted string if followed by a dollar sign, backtick, double quote, another backslash, or a newline; so the \1 is literal, not an escaped 1; reference here).

Short answers:

1. You either single quote the bits that need single quoting (and concatenate that with a double quoted string containing your shell expansions), or you escape what needs to be escaped in a single double quoted string. This is a matter of taste. I'd be more concerned with using a command substitution in the expression, as that is a code injection vulnerability unless you have full control over the string that is inserted.

2. \\\$ or [\$] in a double quoted string. \$ or [$] in a single quoted string.