The window title in a terminal is set by control characters embedded in the output of whatever is running in the terminal. ssh is oblivious to this as it merely passes whatever characters are generated on the remote end back to the client's stdout. Usually most UNIX/Linux systems which "set the window title" do this by embedding the control characters in the shell's prompt
PS1 variable in either in the main
/etc/bash_profile or in the user's default
$HOME/.bash_profile (or whatever configuration file your chosen shell uses). This page describes how to do it in a number of different shells.
So, with that explained, then you log in to a remote machine with this stuff set up, the window title gets set, and the reason the window title doesn't "change back" is probably because your local machine doesn't have these control characters embedded in your local prompt to re-set the window title. In other words, once the window title has been set by these control characters, they don't automatically unset. You have to actually output the control characters again to change it, usually by having them in your local prompt variable.
VIM running on the local machine can query the window title from the X server (by referencing the window ID in
$WINDOWID) and save the title to restore it back when it exits. This can be controlled with the VIM options
:help 'title'). The ssh client simply doesn't seem to do this, again probably because ssh doesn't directly manage the window title at all, it just outputs the characters from the remote shell.
edit: Thanks to Stephane Chazelas in the comments for corrections about VIMs title behavior.