Your two versions:
echo `echo `echo $SHELL``
and
echo $(echo `echo $SHELL`)
are not equivalent. The first backtick command substitution ends as soon as another backtick is seen:
When the old-style backquote form of substitution is used, [...] The first backquote not preceded by a backslash terminates the command substitution.
Your first version is actually equivalent to:
echo $(echo )echo $SHELL$()
which is why you get an "echo" in the output - the command you end up running (after substitutions and with extra whitespace removed) is just:
echo echo /bin/bash
so the output is "echo /bin/bash", just like if you wrote that command out directly.
If you must nest backticks, you can backslash the inner pairs to escape them. Your first command could be written correctly as:
echo `echo \`echo $SHELL\``
although I wouldn't recommend it — $( ... )
is made for nesting.
$()
is for. Backticks don't nest.