I've read in various places, such as in this question that using an ssh key pair without a passphrase allows attackers to steal your private key if they gain access to your account. I assumed that by "steal" it meant they could fairly easily do a brute-force attack to try every possible private key until they found one that matched the public key stored in your account. But the accepted answer on the question above assumes that the private keys are essentially stored in plain text somewhere. This is contrary to what I've always understood to be the case: that since (normally) you wouldn't store your private key on the server--only your public key--the main security risk posed by not using a passphrase is that someone could steal the private key off of your local machine and then use it without needing to also know the passphrase.
Which view is correct? Are private keys somehow easily lifted from a server on which you only placed your public key?