sign_and_send_pubkey: signing failed: agent refused operation

Question

Running xubuntu 16.04, with xfce, I'm trying to use ssh keys with passphrases. I would like to add my passphrased key to my ssh-agent, but I don't know why I can't add it. I don't have gnome keyring enabled or anything alike in my startup. ssh-add privatekey, adds the key but when I try to ssh again it just prints the error two times. Some fixes say to disable the gnome keyring on startup but I've already had it disabled. This all occured when I replaced the ssh keys for my raspberrypi.

> OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/potato/.ssh/config
debug1: /home/potato/.ssh/config line 1: Applying options for paj
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 111.229.105 [111.229.105] port 22253.
debug1: Connection established.
debug1: identity file /home/potato/.ssh/hplaptop_to_pi type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/potato/.ssh/hplaptop_to_pi-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Raspbian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Raspbian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 111.229.105:22253 as 'pi'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve22219-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:zrjeaaHD8TjzsdsdssssA2fXnG3gxp2U
debug1: Host '[111.229.105]:22253' is known and matches the ECDSA host key.
debug1: Found key in /home/potato/.ssh/known_hosts:2
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/potato/.ssh/hplaptop_to_pi
debug1: Server accepts key: pkalg ssh-rsa blen 535
sign_and_send_pubkey: signing failed: agent refused operation
debug1: Offering RSA public key: potato@potato-HP-tomato
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: potato@hplaptop
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: potato@hplaptop
debug1: Server accepts key: pkalg ssh-rsa blen 535
sign_and_send_pubkey: signing failed: agent refused operation
debug1: Offering RSA public key: rsa-key-20141222
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: potato@potatolaptop
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

Answer 1

So after hours of mindless googling and help, the problem was uncovered. I was generating my ssh keys with ssh-keygen and added an additional argument "-o" which generated the keys in a new format for openSSH. The problem was that my gnome-keyring did not support such keys as the keys had Ed255519 signature scheme. Gnome-keyring does not support that since 3.20. I reverted to RSA and no more problems!.