In my case, I had a slightly different problem. When I invoked
ssh-add -D the agent appeared to succeed and responded with
All identities removed. but in fact, when listing the agent keys
ssh-add -l the unwanted keys remained listed, and of course, when attempting to use the agent to authenticate to a remote host the agent would prompt me using my configured pin program for a passphrase to the unwanted keys. Annoying.
The cause of the problem was that my gpg-agent daemon had cached the keys in a file at the path
$ cat ~/.gnupg/sshcontrol
# List of allowed ssh keys. Only keys present in this file are used
# in the SSH protocol. The ssh-add tool may add new entries to this
# file to enable them; you may also add them manually. Comment
# lines, like this one, as well as empty lines are ignored. Lines do
# have a certain length limit but this is not serious limitation as
# the format of the entries is fixed and checked by gpg-agent. A
# non-comment line starts with optional white spaces, followed by the
# keygrip of the key given as 40 hex digits, optionally followed by a
# caching TTL in seconds, and another optional field for arbitrary
# flags. Prepend the keygrip with an '!' mark to disable it.
# RSA key added on: 2021-06-03 16:23:25
# Fingerprints: MD5:c1:[elided]:24
# Ed25519 key added on: 2021-06-03 22:11:36
# Fingerprints: MD5:[elided]:24:da
Deleting those keys from the
~/.gnupg/sshcontrol allowed me to resume use of the gpg-agent to authenticate to remote hosts without the agent demanding a passphrase for keys which I was no longer using.