How to use ssh-agent with multiple (valid) keys and yet choose which one to use?

Question

Let's say there are two keys A and B, both of which are valid for user@host public key authentication. Since authorized_keys is configured for different behaviour depending on the key, ~/.ssh/config on the client uses something like

Host A.host
    HostName host
    User user
    IdentityFile ~/.ssh/A
Host B.host
    HostName host
    User user
    IdentityFile ~/.ssh/B

That works fine. However, the moment I use ssh-agent and add both keys A and B (e.g. in order to enter their passphrases at login instead of when I call the respective ssh A.host or ssh B.host), the connection will always use the same id for both virtual hosts. Is there any way to specify which stored key ssh should use from ssh-agent without having to remove the other key?

Answer 1

As answered elsewhere, the trick is adding the option IdentitiesOnly yes which makes sure that only the configured keys will be used even if others are available from the agent.