It seems that servers are only as secure as the machine that holds the keys and that in turn means the desktop/screensaver password is the line of defence for not only the desktop itself but any remote servers normally accessed.
Is there a way to run a normal disk-based system but always have
secure vnc, and
secure rsync rely on keys from a removable drive system such as a USB flash drive? This way, even if the desktop security is broken (example: stolen desktop or laptop with no password) the servers are safe as long as the USB flash drive is not present.