gpg-agent mysteriously stopped working - agent on remote system no longer connecting to ssh socket

Question

I am using a yubikey nano on my local system to do encrypt/decrypt/sign on remote systems, plus SSH agent forwarding. I recall this being a bear to setup, but it has worked flawlessly for several months now. Suddenly it broke. My searches all return the same links I read when I set it up, but I am stuck.

SSH agent forwarding inexplicably works. Remote system shows this:

REMOTE:$ ssh-add -L
ssh-rsa blahblah cardno:123

I can login to other servers using SSH from the remote system and it uses the nano for auth (I know this because it requires touch to enable agent signing). I can see logs about the SSH signing in the gpg-agent log on the local system.

However, I can't get GPG sign/encrypt to work at all. If I run the following on the remote system:

REMOTE:$ echo "$(uname -a)" |  gpg2 --armor --clearsign --default-key 0x1234
gpg: all values passed to '--default-key' ignored
gpg: no default secret key: No secret key
gpg: [stdin]: clearsign failed: No secret key

In the local gpg-agent log I see no logs about the attempt. If I run this command, I can see log entries in the local gpg-agent log:

REMOTE:$ $ netcat  -U /home/user/.gnupg/S.gpg-agent
OK Pleased to meet you
RESET
OK
GETINFO PID
ERR 67109115 Forbidden <GPG Agent>
POOP
ERR 67109139 Unknown IPC command <GPG Agent>

Which results in these logs in the local agent:

2018-01-05 16:38:32 gpg-agent[865] DBG: chan_10 -> OK Pleased to meet you
2018-01-05 16:38:35 gpg-agent[865] DBG: chan_10 <- RESET
2018-01-05 16:38:35 gpg-agent[865] DBG: chan_10 -> OK
2018-01-05 16:38:45 gpg-agent[865] DBG: chan_10 <- GETINFO PID
2018-01-05 16:38:45 gpg-agent[865] DBG: chan_10 -> ERR 67109115 Forbidden <GPG Agent>
2018-01-05 16:39:01 gpg-agent[865] DBG: chan_10 <- POOP
2018-01-05 16:39:01 gpg-agent[865] DBG: chan_10 -> ERR 67109139 Unknown IPC command <GPG Agent>

If I run strace -f -F on gpg-connect-agent on the remote system, it seems to be connecting to a socket in /var/run, but not the one forwarded from the local system in ~/.gnupg/. I have tried removing both sockets, killing all gpg-agent processes and changed the SSH remote forward to go to either the /var/run location or the ~/.gnupg location to no avail. It is possible I screwed these steps up and I will try them again, but I want to know if someone knows the answer, and I would like to have an easy to find post for the next time this breaks.

LOCAL SYSTEM:

Mac OS X 10.11.6
gpg installed with brew
gpg (GnuPG) 2.2.1
libgcrypt 1.8.1

REMOTE SYSTEM:

ubuntu 17.10
gpg (GnuPG) 2.1.15
libgcrypt 1.7.8

EDIT: Ok, no idea what changed, but I left it alone for a bit and came back and tried to switch the socket again and it now works:

REMOTE:$ $ echo "$(uname -a)" |  strace -f -F gpg2 --armor --clearsign --default-key 0x1234
...
a bunch of garbage
...
stat("/run/user/1000/gnupg/S.gpg-agent", {st_mode=S_IFSOCK|0600, st_size=0, ...}) = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 5

Changing my SSH remote forward to this new location worked. I swear I tried this earlier using the socket path provided by gpgconf --list-dir agent-ssh-socket, without any luck. Probably forgot to kill the existing agent. And by happenstance, I just chanced upon a blogpost reporting that this changed: https://blog.kylemanna.com/linux/gpg-213-ssh-agent-socket-moved/

Answer 1

UPDATE: UNLINK YOUR SOCKET, See the edit at the bottom.

There is clearly a bug here that is intermittent or this is a side effect of how all these systems interact together. I don't want to go into too much detail/conjecture, but I would consider the following:

• Second SSH session hoses pipes of the first (should that happen?)
• StreamLocalBindUnlink in SSHD on REMOTE is not reaping pipes properly (possibly because agent on remote end still has them open?)

A great workaround would be to use a separate and special host in .ssh/config for gpg agent forwarding, and make sure to use login to that hostname only once!

The few times it has happened were very frustrating because the combination of pipes/running agents and instances where a new sshd session did not create new sockets led to all sorts of confusion (again possibly due to old socket files being held open by a running agent). And usually, I don't have time to be trying to fix such a problem.

Having finally encountered this where I was in a position to troubleshoot and fix, I can reliably reproduce the problem and show how to fix it:

Working System

REMOTE:$ echo "$(uname -a)" | gpg --armor --clearsign --default-key 0x1234
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Linux pooter 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEE/0lJ/t51agRvvTILdQ2kyDf6wDYFAl9JgSYACgkQdQ2kyDf6

-----END PGP SIGNATURE-----

Run another SSH or Rsync from the local system

LOCAL: $ rsync -avze ssh test.txt pooter:
bind [127.0.0.1]:5901: Address already in use

SCP doesn't cause a problem, but RSYNC clearly does an ssh login as I see port forwards fail as the file copies over

Remote encrypt/decrypt now fails

REMOTE:$ echo "$(uname -a)" | gpg --armor --clearsign --default-key 0x1234
gpg: all values passed to '--default-key' ignored
gpg: no default secret key: No secret key
gpg: [stdin]: clear-sign failed: No secret key

The Fix

1. Log out all SSH sessions that have socket forwards
2. Kill all gpg agents on the REMOTE system
3. Verify pipes in /var/run/xxxx/gnupg are in place
4. If pipes go away, login with socket forward and verify they get recreated

For steps 3 and 4 I have seen it go both ways, sometimes the pipes remain, and sometimes they properly go away. It may also be necessary to remove the pipe files and log back in and make sure they get recreated.

Everything should now work again for encryption/decryption

UPDATE - SIMPLER FIX

Today this happened for no rhyme or reason, and I was quite annoyed. I had to come find this post and I was too lazy to do everything my original post was telling me to do for the fix. I considered the situation from the perspective of my conjectures above. Long story short, this worked:

1. killall gpg-agent && exit
2. CTRL^C # To disconnect forwarded ports
3. Wait ~1 minute
4. Log back in

Everything was working again

EDIT: I had this happen again and I added one more step. Find your unix domain socket file and run the following from a different shell/login: unlink /path/to/socket