I had the same problem as you did. I'm not an expert, but the solution that worked for me was to remove the keys from the ~/.ssh directory. This is an example of what I was seeing.
$ ssh-add -l
2048 06:e9:a6:14:2a:e4:c3:11:56:ea:c3:5d:f9:84:79:c6 first key (RSA)
2048 2c:c3:97:fe:f3:cf:03:dc:d3:0b:87:2b:01:72:33:3b second key (RSA)
$ ssh-add -d ~/.ssh/id_rsa.pub
Identity removed: /home/user/.ssh/ybot_rsa.pub (/home/user/.ssh/id_rsa.pub)
$ ssh-add -l
2048 06:e9:a6:14:2a:e4:c3:11:56:ea:c3:5d:f9:84:79:c6 first key (RSA)
2048 2c:c3:97:fe:f3:cf:03:dc:d3:0b:87:2b:01:72:33:3b second key (RSA)
It was after moving id_rsa (and id_rsa.pub) from the ~/.ssh folder that the ssh-agent stopped having the identity. What I did was:
$ ssh-add -l
2048 06:e9:a6:14:2a:e4:c3:11:56:ea:c3:5d:f9:84:79:c6 first key (RSA)
2048 2c:c3:97:fe:f3:cf:03:dc:d3:0b:87:2b:01:72:33:3b second key (RSA)
$ mv ~/.ssh/id_rsa ~/.ssh/id_rsa.pub ~/
$ ssh-add -l
2048 2c:c3:97:fe:f3:cf:03:dc:d3:0b:87:2b:01:72:33:3b second key (RSA)
I'm guessing (I could be wrong) that if you didn't have the Gnome Keyring daemon running (gnome-keyring-daemon), this behaviour would change. Maybe the original (and logical) solution would work, but not the second one.
openssh
6.2p1-1pacman -Ss gnome-keyring: 3.6.3-1