It seems that servers are only as secure as the machine that holds the keys and that in turn means the desktop/screensaver password is the line of defence for not only the desktop itself but any remote servers normally accessed.
Is there a way to run a normal disk-based system but always have ssh, scp, secure vnc, and secure rsync rely on keys from a removable drive system such as a USB flash drive? This way, even if the desktop security is broken (example: stolen desktop or laptop with no password) the servers are safe as long as the USB flash drive is not present.


ssh-agent(in various forms). – sr_ Dec 14 '12 at 14:43