Why does the exclamation mark `!` sometimes upset bash?

Question

I realize that ! has special significance on the commandline in the context of the commandline history, but aside from that, in a runing script the exclamation mark can sometimes cause a parsing error.
I think it has something to do with an event, but I have no idea what an event is or what it does.  Even so, the same command can behave differently in different situations.
The last example, below, causes an error; but why, when the same code worked outside of the command substitution? .. using GNU bash 4.1.5

# This works, with or without a space between ! and p
  { echo -e "foo\nbar" | sed -nre '/foo/! p'
    echo -e "foo\nbar" | sed -nre '/foo/!p'; }
# bar
# bar

# This works, works when there is a space between ! and p
  var="$(echo -e "foo\nbar" | sed -nre '/foo/! p')"; echo "$var"
# bar

# This causes an ERROR, with NO space between ! and p
  var="$(echo -e "foo\nbar" | sed -nre '/foo/!p')"; echo "$var"
# bash: !p': event not found

Answer 1

The ! character invokes bash's history substitution (enabled by default in interactive shells). When followed by a string (as in your failing example) it tries to expand to the last history event that began with that string. Just like $var gets expanded to the value of that string, !echo would expand to the last echo command in your history.

Space is a breaking character in such expansions. First note how this would work with variables:

# var="like"
# echo "$var"
like
# echo "$"
$
# echo "Do you $var frogs?"
Do you like frogs?       <- as expected, variable name broken at space
# echo "Do you $varfrogs?"
Do you?                  <- $varfrogs not defined, replaced with blank
# echo "Do you $ var frogs?"
Do you $ var frogs?      <- $ not a valid variable name, ignored

The same thing will happen for history expansion. The bang character (!) starts off a history replacement sequence, but only if followed by a string. Following it with a space make it literal bang instead of part of a replace sequence.

You can avoid this kind of replacement for both variable and history expantion by using single quotes. Your first examples used single quotes and so ran fine. Your last examples are in double quotes and thus bash scanned them for expantion sequences before it did anything else. The only reason the first one didn't trip is that that the space is a break character as shown above.

Answer 2

As already said by Caleb, ! is used to invoke bash's history substitution.

If like me you feel you don't need such a feature, you can disable it inserting the following line in ~/.bashrc:

set +H

I don't need it because the history can be recovered by the up arrow and Ctrl-r incremental reverse search. See bash's manual page, section Commands for Manipulating the History for a detailed list of shortcuts.

Answer 3

your first example:

{ echo -e "foo\nbar" | sed -nre '/foo/! p'
    echo -e "foo\nbar" | sed -nre '/foo/!p'; }

could be reduced to

echo '! p' 
echo '!p'

Within single quotes, all characters preserve their literal values. Thus ! has lost its special meaning and history expansion is not preformed.

your second and third examples:

var="$(echo -e "foo\nbar" | sed -nre '/foo/! p')"; echo "$var"

var="$(echo -e "foo\nbar" | sed -nre '/foo/!p')"; echo "$var"

could be reduced to

echo "'! p'"

echo "'!p'"

'! p' and '!p' are essentially parts of the double quoted strings.

Within double quotes, all characters preserve their literal values except $, `, \ and !.

That implies the single quotes of '! p' and '!p' have lost their special meanings (i.e.: unable to escape !) but ! still retains its special meaning thus history expansion is performed.

However, when ! is followed by a space character, history expansion is not performed.

Quoting from man bash:

QUOTING

[...]

Enclosing characters in single quotes preserves the literal value of each character within the quotes. [...]

Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !. [...] If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.

HISTORY EXPANSION

[...]

History expansions are introduced by the appearance of the history expansion character, which is ! by default. Only backslash (\) and single quotes can quote the history expansion character.

Several characters inhibit history expansion if found immediately following the history expansion character, even if it is unquoted: space, tab, newline, carriage return, and =. If the extglob shell option is enabled, ( will also inhibit expansion.