1

I have a reverse SSH setup to traverse an impenetrable CGN NAT. It works well, but the SSHD daemon on the client behind the NAT keeps dieing for unknown reasons (fails to start-up). The client successfully makes reverse SSH connections to my server (w/ public-private keypair), but once i try to reverse SSH connect to the NATed client, i get:


Working:

user@server:~$ ssh user@127.0.0.1 -p 22424
user@127.0.0.1's password:

Linux NATed-client 3.12.22+ #691 PREEMPT Wed Jun 18 18:29:58 BST 2014 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Dec 13 16:41:16 2014 from localhost

user@NATed-client ~ $

Defective:

user@server:~$ ssh user@127.0.0.1 -p 22424
ssh_exchange_identification: Connection closed by remote host

I have this Reverse SSH script set to run in the NATed Client's crontab every 15 minutes as a standard user:

#!/bin/bash
createTunnel() {
    /usr/bin/ssh -i /home/ssh_user_NATed_client/.ssh/NATed_Client_key \
        -N -R 22424:localhost:22 ssh_user_server@server.com \
        -o ExitOnForwardFailure=yes 
    if [[ $? -eq 0 ]]; then
        echo Tunnel created
    else
        echo An error occurred creating a tunnel. RC was $?
    fi
}
/bin/pidof ssh
if [[ $? -ne 0 ]]; then
    echo Creating new tunnel connection
    createTunnel
fi

I've opened the sshd_config file on the NATed client after this happened, and it is unintelligible gibberish. When I replace it with a backup sshd_config, everything works again. It's as if the sshd on the NATed client chewed up the config file randomly and then the sshd cannot start (leading to the SSH sessions requested on the reverse SSH connection to close like this). This has happened 3 times over the last 6 months. The file permission are:

$ ls -l ssh_config
-rw-r--r-- 1 root root   1669 Feb  8  2013 ssh_config

The NATed client computer is still running. It establishes the reverse SSH connection every 15 minutes via crontab and other automatic functions I've programmed still work. I just can't control it or SSH back into it on the reverse connection. I'm pretty sure the SSH host/server on the NATed client has died/refuses to start at bootup.

I'll point out that it's a raspberry pi running on a 32GB SD card. It's been running for about 6 months. Might possibly be flash wear, but there are no kernel panics (very rarely write to flash anyway).

Any suggested workarounds or secondary SSH backdoors I can put on this system? These random sshd deaths are killing me.

Thank you for your insight.

  • ssh_config you list (I suppose in /etc/ssh is not that ineresting - you want sshd_config you mentioned a couple of lines earlier. Apart from that, you can mark the sshd_config as immutable, which will prevent any changes to it. Use chattr +i /etc/ssh/sshd_config, once you have the right configuration in place. If you spot rubbish there later on, at least one of your 1) file system, 2) kernel and 3) hardware is broken. Check system logs. Try dropbear, if you want an alternative to OpenSSH. – peterph Dec 16 '14 at 18:10
  • i meant sshd_config. Thanks for the clarification. do you think dropbear can co-exist with this reverse ssh solution as well? Other IP services ride on these reverse SSH links, so I cannot abandon the current solution. Just need a 2nd remote terminal solution to recover from this currently unrecoverable error. – etronz Dec 16 '14 at 18:17
  • What happens if you simply make /etc/ssh/sshd_config immutable? sudo chattr +i /etc/ssh/sshd_config? BTW, your crontab system to keep the tunnel open is clunky, you should use the more sophisticated, autossh, it exists for Raspbian as well (I am using it). – MariusMatutiae Dec 16 '14 at 18:34
  • I will attempt immutable in my testbed pi when i get a chance. The Reverse SSH link to the client's SSHD is down /w this error, so cannot do anything right now. I'm going to swap out SD cards w/ a new solution based upon what's discovered from this thread's discussion. I appreciate the suggestions for dropbear, autossh, and immutable. I read about autossh, it looks like an excellent platform to nail up these connections. I couldn't get it to work on my first attempts, so deployed the clunky crontab configuration instead. – etronz Dec 16 '14 at 20:28
  • Update: I've abandoned my prior implementation and went with AutoSSH. I also made the sshd_config file immutable. The new reverse connections are much more stable. Thank you for your suggestions. – etronz Aug 8 '15 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.