Is it *really* possible to directly steal a private key if it uses no passphrase?

Question

I've read in various places, such as in this question that using an ssh key pair without a passphrase allows attackers to steal your private key if they gain access to your account. I assumed that by "steal" it meant they could fairly easily do a brute-force attack to try every possible private key until they found one that matched the public key stored in your account. But the accepted answer on the question above assumes that the private keys are essentially stored in plain text somewhere. This is contrary to what I've always understood to be the case: that since (normally) you wouldn't store your private key on the server--only your public key--the main security risk posed by not using a passphrase is that someone could steal the private key off of your local machine and then use it without needing to also know the passphrase.

Which view is correct? Are private keys somehow easily lifted from a server on which you only placed your public key?

Answer 1

That question is worried about private keys stored on the server unencrypted. It's a scenario like:

workstation 1 ---> gateway -> final server
     ⋮               |
workstation n ------/

and the OP is worried about private keys on "gateway", which is a shared machine with multiple users.

It is not possible to steal the private key by compromising the server, only the machine the key is actually stored on. If you use ssh agent forwarding, then its possible to use the key without stealing it (to log into another machine, for example), if you do not have ssh-agent prompt first.

But you should still encrypt your private keys, in case your workstation is compromised. Or, especially in the case of a laptop, lost or stolen.