`man bash`, search for section _Special Parameters_:

> $_ At shell startup, set to the absolute pathname used to invoke the
> shell or shell script being executed as passed in the  environment  or
> argument list.  Subsequently, expands to the last argument to the
> previous command, after expansion.  Also set to the full pathname used
> to invoke each command executed and placed in the environment exported
> to that command.  When  checking mail, this parameter holds the name
> of the mail file currently being checked.

The key difference is the expansion. So if I have environment variable `FOO` set to 'bar' and I do `echo $FOO` then `$_` on the next line will display `bar` while `!$` will just display `$FOO`