After a few hours of digging, I finally found out why my computer stopped asking me for my key(s) using gnome ssh-askpass. It was very simple, but took me forever to figure out because... well... most of the people who are having similar problems is for a completely different reason (i.e. ssh-agent not running).

In my case, I had too many keys under ~/.ssh so what I had done was to move them all, except id_rsa in a sub-directory: ~/.ssh/other_keys. That worked for the purpose of accessing some computers just using the -i command line option as in:

ssh -i ~/.ssh/other_keys/topsecret domain

Otherwise you have to tell SSH to not test all the keys found in your ~/.ssh directory (because when that happens, it exhausts the number of tries which is usually pretty small like 3 or 5.)

To fix this problem you could also do this:

ssh -i ~/.ssh/topsecret -o 'IdentitiesOnly yes' domain

Which is a lot of typing (yes, you can also add it to your config file, but if that's just for a quick test, that's an annoyance.)

So... all of that works great for when I want to connect once here or once there, but it is painful when I want to go to one of my main servers because ssh now asks me for my passphrase on each access, even though my ssh-agent and ssh-askpass are properly setup.

So... what gives?

Somehow, when you start ssh-add it automatically adds the keys to the ssh-agent without asking you for your passphrases. Then once you need to use the key, it automatically starts your ssh-askpass (no need for a variable of that name in your environment, btw) where you can enter your passphrase and move on.

If the keys are defined in a sub-directory, however, they do not get found so they do not get added to the ssh-agent and when you use ssh it does not find them in the ssh-agent either and thus asks you for your passphrase directly in your console and not in a gnome popup window.

Moving keys back directly under ~/.ssh fixes the problem, but that's not really a good solution since it breaks the ssh -i ... again. (I know, there is the -o ... option to tell ssh to use that one key only and that resolves that other problem. But I think ssh should test with the command line specified key FIRST and that would resolve both my problems. Oh well...)

So... when I run the following command:

ssh-add -D

It resets the ssh-agent keys by (1) deleting all the keys currently there; and (2) reinstalling all the keys found under ~/.ssh without asking me for any passphrase (at least not at the time I run the ssh-add command).

Would there be a way to ask ssh-add (or whatever other tool) to also add the keys found in a different directory such as my ~/.ssh/other_keys sub-directory and this without having to enter all the passphrases right then?

  • 3
    FYI, you can work around the issue by specifying which keys go with which sites in your $HOME/.ssh/config file too. This eliminates ssh from having to search through the keys present in the user's $HOME/.ssh directory.
    – slm
    Commented Dec 1, 2014 at 0:29
  • Would it be an option to use gpg-agent instead? Commented Dec 1, 2014 at 6:38
  • @HaukeLaging, Well... I was more hoping for not additional setup. Especially because the default ssh-agent gets installed and setup automatically in Ubuntu. Commented Dec 1, 2014 at 8:06

2 Answers 2


Not a full blown answer but if you look in the man page of ssh-agent it has the following paragraph.

The agent initially does not have any private keys. Keys are added using ssh-add(1). When executed without arguments, ssh-add(1) adds the files ~/.ssh/id_rsa, ~/.ssh/id_dsa, ~/.ssh/id_ecdsa and ~/.ssh/identity. If the identity has a passphrase, ssh-add(1) asks for the passphrase on the terminal if it has one or from a small X11 program if running under X11. If neither of these is the case then the authentication will fail. It then sends the identity to the agent. Several identities can be stored in the agent; the agent can automatically use any of these identities. ssh-add -l displays the identities currently held by the agent.

So you can see what keys are currently loaded by ssh-agent and it will default to loading the keys as described by that paragraph.

  • That's the thing, it actually adds any private key that has a corresponding .pub file in ~/.ssh without asking you for your passphrase, but it does not want to add other keys in a similar fashion. Commented Dec 1, 2014 at 2:30
  • @AlexisWilke - try importing them via seahorse instead.
    – slm
    Commented Dec 1, 2014 at 3:31

Maybe the AddKeysToAgent option (available in recent OpenSSH versions) could be useful in this situation?

Specifies whether keys should be automatically added to a running ssh-agent(1). If this option is set to yes and a key is loaded from a file, the key and its passphrase are added to the agent with the default lifetime, as if by ssh-add(1). If this option is set to ask, ssh(1) will require confirmation using the SSH_ASKPASS program before adding a key (see ssh-add(1) for details). If this option is set to confirm, each use of the key must be confirmed, as if the -c option was specified to ssh-add(1). If this option is set to no, no keys are added to the agent. The argument must be yes, confirm, ask, or no (the default).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .