The SSH protocol is defined by what the
sshd programs accept. (There is a standard defined for it, but it's an after-the-fact thing and is mostly ignored when one of the implementations adds new features.) Since there are multiple implementations of those (OpenSSH, F-Secure, PuTTY, etc.) occasionally you'll find that one of them doesn't support the same protocol as the others. Basically, it defines authentication negotiation and creation of a multiplexed data stream. This stream can carry one or more (with OpenSSH and
ControlMaster) terminal sessions and zero or more tunnels (forwarding socket connections from either local or remote to the other side; X11 forwarding is a special case of remote forwarding). It also defines "subsystems" that can be used over the stream; terminal sessions are the basic subsystem but others can be defined.
sftp is one of these.
ssh the utility uses the SSH protocol to talk to
sshd on another machine. How it works depends on what version it is (see above), but the gist of it is that it attempts to figure out which version of the SSH protocol to use, then it and
sshd negotiate supported authentication methods, then it tries to authenticate you using one of those methods (asking for remote user password/private key paasword/S-Key phrase as necessary), and on successful authentication sets up a multiplexed stream with the
sshd, as said above, implements the server side of the SSH protocol.
sftp is a (at present, the only standard) subsystem defined in most
sshd implementations. When the SFTP subsystem is requested,
sftp-server to the subsystem session; the
sftp program then talks to it, similarly to
ftp but with file transfers multiplexed on the stream instead of using separate connections as with
su has nothing to do with
sftp, except insofar as there may be PAM modules to arrange for the multiplexed stream to be available within the shell or program run by it.