5

I am using a yubikey nano on my local system to do encrypt/decrypt/sign on remote systems, plus SSH agent forwarding. I recall this being a bear to setup, but it has worked flawlessly for several months now. Suddenly it broke. My searches all return the same links I read when I set it up, but I am stuck.

SSH agent forwarding inexplicably works. Remote system shows this:

REMOTE:$ ssh-add -L
ssh-rsa blahblah cardno:123

I can login to other servers using SSH from the remote system and it uses the nano for auth (I know this because it requires touch to enable agent signing). I can see logs about the SSH signing in the gpg-agent log on the local system.

However, I can't get GPG sign/encrypt to work at all. If I run the following on the remote system:

REMOTE:$ echo "$(uname -a)" |  gpg2 --armor --clearsign --default-key 0x1234
gpg: all values passed to '--default-key' ignored
gpg: no default secret key: No secret key
gpg: [stdin]: clearsign failed: No secret key

In the local gpg-agent log I see no logs about the attempt. If I run this command, I can see log entries in the local gpg-agent log:

REMOTE:$ $ netcat  -U /home/user/.gnupg/S.gpg-agent
OK Pleased to meet you
RESET
OK
GETINFO PID
ERR 67109115 Forbidden <GPG Agent>
POOP
ERR 67109139 Unknown IPC command <GPG Agent>

Which results in these logs in the local agent:

2018-01-05 16:38:32 gpg-agent[865] DBG: chan_10 -> OK Pleased to meet you
2018-01-05 16:38:35 gpg-agent[865] DBG: chan_10 <- RESET
2018-01-05 16:38:35 gpg-agent[865] DBG: chan_10 -> OK
2018-01-05 16:38:45 gpg-agent[865] DBG: chan_10 <- GETINFO PID
2018-01-05 16:38:45 gpg-agent[865] DBG: chan_10 -> ERR 67109115 Forbidden <GPG Agent>
2018-01-05 16:39:01 gpg-agent[865] DBG: chan_10 <- POOP
2018-01-05 16:39:01 gpg-agent[865] DBG: chan_10 -> ERR 67109139 Unknown IPC command <GPG Agent>

If I run strace -f -F on gpg-connect-agent on the remote system, it seems to be connecting to a socket in /var/run, but not the one forwarded from the local system in ~/.gnupg/. I have tried removing both sockets, killing all gpg-agent processes and changed the SSH remote forward to go to either the /var/run location or the ~/.gnupg location to no avail. It is possible I screwed these steps up and I will try them again, but I want to know if someone knows the answer, and I would like to have an easy to find post for the next time this breaks.

LOCAL SYSTEM:

Mac OS X 10.11.6
gpg installed with brew
gpg (GnuPG) 2.2.1
libgcrypt 1.8.1

REMOTE SYSTEM:

ubuntu 17.10
gpg (GnuPG) 2.1.15
libgcrypt 1.7.8

EDIT: Ok, no idea what changed, but I left it alone for a bit and came back and tried to switch the socket again and it now works:

REMOTE:$ $ echo "$(uname -a)" |  strace -f -F gpg2 --armor --clearsign --default-key 0x1234
...
a bunch of garbage
...
stat("/run/user/1000/gnupg/S.gpg-agent", {st_mode=S_IFSOCK|0600, st_size=0, ...}) = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 5

Changing my SSH remote forward to this new location worked. I swear I tried this earlier using the socket path provided by gpgconf --list-dir agent-ssh-socket, without any luck. Probably forgot to kill the existing agent. And by happenstance, I just chanced upon a blogpost reporting that this changed: https://blog.kylemanna.com/linux/gpg-213-ssh-agent-socket-moved/

7
  • I found the solution. I don't seem to have enough fake internet points to do anything further. Should I just delete the question? It has enough debugging pointers that it might be of use to someone in the future.
    – lopass
    Jan 6, 2018 at 4:06
  • I put it up for a close vote.
    – B Layer
    Jan 6, 2018 at 4:17
  • This still is happening to me on occasion. I believe the trigger is when I am connected to the remote computer in a shell and I run SCP or RSYNC to the remote computer in another terminal. I know from experience that logging out it will fix itself overnight, but I have not been able to find the problem or how to fix it. Thinking about asking another question to see if anyone knows
    – lopass
    Sep 26, 2019 at 16:17
  • I asked this question again a while back, but the other question only has 19 views while this has 1000. So I am adding this info here. Problem keeps happening and I seem to finally have found a solution: 1. Log out all ssh sessions 2. Use terminal access to kill all gpg agents 3. Check that /run/user/XXXX/gnupg sockets are intact 4. Log in with SSH again (note this has all of the socket forwards as per docs) Everything should work again. I will update with another comment if I find anything different
    – lopass
    Aug 26, 2020 at 16:38
  • If this is a non-trivial, reproducible problem, contrary to what we thought two years ago, then it should probably be reopened and you should post an answer if you have a legit solution.
    – B Layer
    Aug 26, 2020 at 19:04

1 Answer 1

0

There is clearly a bug here that is intermittent or this is a side effect of how all these systems interact together. I don't want to go into too much detail/conjecture, but I would consider the following:

  • Second SSH session hoses pipes of the first (should that happen?)
  • StreamLocalBindUnlink in SSHD on REMOTE is not reaping pipes properly (possibly because agent on remote end still has them open?)

A great workaround would be to use a separate and special host in .ssh/config for gpg agent forwarding, and make sure to use login to that hostname only once!

The few times it has happened were very frustrating because the combination of pipes/running agents and instances where a new sshd session did not create new sockets led to all sorts of confusion (again possibly due to old socket files being held open by a running agent). And usually, I don't have time to be trying to fix such a problem.

Having finally encountered this where I was in a position to troubleshoot and fix, I can reliably reproduce the problem and show how to fix it:

Working System

REMOTE:$ echo "$(uname -a)" | gpg --armor --clearsign --default-key 0x1234
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Linux pooter 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEE/0lJ/t51agRvvTILdQ2kyDf6wDYFAl9JgSYACgkQdQ2kyDf6

-----END PGP SIGNATURE-----

Run another SSH or Rsync from the local system

LOCAL: $ rsync -avze ssh test.txt pooter:
bind [127.0.0.1]:5901: Address already in use

SCP doesn't cause a problem, but RSYNC clearly does an ssh login as I see port forwards fail as the file copies over

Remote encrypt/decrypt now fails

REMOTE:$ echo "$(uname -a)" | gpg --armor --clearsign --default-key 0x1234
gpg: all values passed to '--default-key' ignored
gpg: no default secret key: No secret key
gpg: [stdin]: clear-sign failed: No secret key

The Fix

  1. Log out all SSH sessions that have socket forwards
  2. Kill all gpg agents on the REMOTE system
  3. Verify pipes in /var/run/xxxx/gnupg are in place
  4. If pipes go away, login with socket forward and verify they get recreated

For steps 3 and 4 I have seen it go both ways, sometimes the pipes remain, and sometimes they properly go away. It may also be necessary to remove the pipe files and log back in and make sure they get recreated.

Everything should now work again for encryption/decryption

UPDATE - SIMPLER FIX

Today this happened for no rhyme or reason, and I was quite annoyed. I had to come find this post and I was too lazy to do everything my original post was telling me to do for the fix. I considered the situation from the perspective of my conjectures above. Long story short, this worked:

  1. killall gpg-agent && exit
  2. CTRL^C # To disconnect forwarded ports
  3. Wait ~1 minute
  4. Log back in

Everything was working again

3
  • Why do you say echo "$(uname -a)" | … instead of uname -a | …? Aug 29, 2020 at 3:14
  • that is in a script I use to test remote encryption circa when I first set it up. I probably copy/pasted from the setup instructions. I agree it is superfluous, but it works.
    – lopass
    Aug 30, 2020 at 3:32
  • I had to manually remove the pipe from the remote, then relogin with ssh
    – atevm
    Jul 15, 2021 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.