2

I have two laptops and two Raspberry Pi's on my network. The issue is, I can't ssh into one RPi from any of the machines, neither can I ssh into other machines from that RPi. Both Pi's have a fresh install of Raspbian Buster Lite. I can successfully ping that Pi, however ssh connection times out. My sshfs is not working too, as expected. I have tried to purge ssh, openssh-client and openssh-server, and install them again, however haven't had any success. The debug output isn't very helpful or descriptive:

$ ssh pi@192.168.0.15 -vvv
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "192.168.0.15" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 192.168.0.15 [192.168.0.15] port 22.
debug1: connect to address 192.168.0.15 port 22: Connection timed out
ssh: connect to host 192.168.0.15 port 22: Connection timed out

All my machines are on 192.168.0.*:

What can I do in order to overcome the issue?

@EDIT:

I could at one point ssh into RPi from one laptop, but not from the other. Now I can't do it from either of them.

nmap port scan output:

$ nmap -p 22 192.168.0.15

Starting Nmap 7.60 ( https://nmap.org ) at 2019-12-17 17:05 CET
Nmap scan report for 192.168.0.15
Host is up (0.010s latency).

PORT   STATE    SERVICE
22/tcp filtered ssh

Nmap done: 1 IP address (1 host up) scanned in 0.43 seconds

No firewall running on RPi's.

$ sudo netstat --tcp --programs --listening --numeric
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1335/sshd           
tcp6       0      0 :::22                   :::*                    LISTEN      1335/sshd  

As per request:

$ sudo ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:ab:f8:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.15/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ba27:ebff:feab:f8fb/64 scope link 
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b8:27:eb:fe:ad:ae brd ff:ff:ff:ff:ff:ff
$ sudo ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.15  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::ba27:ebff:feab:f8fb  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:ab:f8:fb  txqueuelen 1000  (Ethernet)
        RX packets 121039  bytes 19525595 (18.6 MiB)
        RX errors 0  dropped 11  overruns 0  frame 0
        TX packets 26795  bytes 5950306 (5.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2  bytes 78 (78.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2  bytes 78 (78.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ssh config files:

broken Pi:

$ cat /etc/ssh/ssh_config

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Protocol 2
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes

$ cat /etc/ssh/sshd_config
#   $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem   sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#   X11Forwarding no
#   AllowTcpForwarding no
#   PermitTTY no
#   ForceCommand cvs server

working Pi:

$ cat /etc/ssh/ssh_config 

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Protocol 2
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
$ cat /etc/ssh/sshd_config 
#   $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem   sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#   X11Forwarding no
#   AllowTcpForwarding no
#   PermitTTY no
#   ForceCommand cvs server
  • Firewall running anywhere? – Panki Dec 17 '19 at 15:10
  • @Panki I've updated the question. – adder Dec 17 '19 at 15:35
  • 1
    Just to be sure: ssh is not enabled by default on vanilla installs. Did you enable it? Could you also run a port scan? nmap -p 22 <pi-address> – Fiximan Dec 17 '19 at 15:38
  • Yes it is enabled, I ran sudo systemctl enable ssh and sudo service ssh start. See the updated question for port scan output. – adder Dec 17 '19 at 16:06
  • To exclude the error coming from your network: can you ssh from one laptop to the other and vice versa? What do the port scans from a Pi on a laptop or laptop A on laptop B show? – Fiximan Dec 17 '19 at 17:31
1

I still cannot write comments but according to nmap documentation:

Filtered means that a firewall, filter, or other network obstacle is blocking the port so that Nmap cannot tell whether it is open or closed. Closed ports have no application listening on them, though they could open up at any time. Ports are classified as unfiltered when they are responsive to Nmap's probes, but Nmap cannot determine whether they are open or closed. Nmap reports the state combinations open|filtered and closed|filtered when it cannot determine which of the two states describe a port.

My guess is that the router has blocked the port somehow and maybe the working other raspi is an error - maybe you can check this one and make sure to restart the router/switch if you changed something or in general.

Also ruling out a hardware error by interchanging modules might help - meaning using the wall socket/power supply unit from the working one (should have enough Amps), putting the sdcard with buster from the working one in the not working one and the other way around (had that one time - a defect network adapter).

Also do you use password verification or an rsa file?

What content has the /etc/hosts file on both of your pies, the working one and the one with the ssh issue? (cat /etc/hosts)

Also please the contents of /etc/ssh/ssh_config (client config) and /etc/ssh/sshd_config (ssh server config) of the pi that doesn*t work and potential differentiation to the working raspberry pi. (cat /etc/ssh/ssh_config && cat /etc/sshd_config)


Edit: Further try

Since the internet works (internet runs through port 80, 8080 and 443) we can try the following thing on the "broken" pi:

  1. uncomment # Port 22 in /etc/ssh/sshd_conf and change it to Port 80 If that doesn't work change it to Port 443 and Port 8080 since that ports should be available. (you change it f.e. with sudo nano /etc/ssh/sshd_conf and safe the changes with ctrl + x)

  2. Than sudo systemctl restart sshd.service between each change and try to connect to the broken pie again from another device with ssh -p 80 username@192.168.1.15 between each change.

-p stands for the port variable and since you changed the port and restarted the adapter with systemctl (you can also restart the pi each time if you want to make sure).

  1. Change the port back to Port 22 if that still doesn't work. (Just fyi: If you have a public server with ssh enabled you should change the port to anything uncommon to make random attacks less easy like port 2409 or something)

  2. You also can check the daemon status - if there are errors or "failed", not loaded errors on the output of the broken pie with sudo systemctl status sshd.service please also post them.

  3. I guess its not easy to access the hardware, but if its not too much trouble I would really recommend interchanging the hardware even if internet works and it sounds stupid - just to rule it out. I had the craziest hardware issues - electrically it is such a cascade in signals and one modulator module can do unexpected behavior - got the funniest things over the years and since "it worked and than it doesn't" makes a hardware defect really possible.

  4. Your ssh should already working through another port IF thats due to a port. If not, a further step is: Did you do an updates (f.e. sudo apt update && sudo apt upgrade) or installed anything right before id didn't worked? Can you remember anything, the smallest change before it doesn't worked? Also did you do a recent update (sudo apt update && sudo apt upgrade -y) - if not, do it now and restart the pi - sometimes even that helps.

| improve this answer | |
  • I don't think it's a hardware issue, the internet is working fine through the Ethernet, and also I could ssh into the broken Pi up to some point just fine. Then, all of a sudden, something went wrong. So I think we can rule that out. When it comes to the router, when I log in to check, there are no restricted ports under Security > Firewall, so I think it's not that. Again, it used to work just fine. I'll post the contents of config files in a minute, check updated question. – adder Dec 17 '19 at 19:43
  • @dedmauz69 I edited my answer with a second section. If its really a port problem, that should fix it. If not it is not a port problem and we have to rule out further stuff. – Ivanovic Dec 17 '19 at 21:16
  • Also: do you use a password to log into the ssh or do you use a passphrase to log in via ssh without a password? It can also worth a try to change 192.168.0.15 to something else like 192.168.0.2 or what ever is open/unused. The lower the better, maybe DHCP is interfering, got that one time. – Ivanovic Dec 17 '19 at 21:26
  • @dedmauz69 Am really curious if you got your problem solved and how. Please don't forget to mark the question as solved if you did else it still shows up in the question s section – Ivanovic Dec 23 '19 at 18:47
0
PORT   STATE    SERVICE
22/tcp filtered ssh

ssh is blocked in one or another way:

  • on PI box by a firewall;
  • you use a smart router which may use a security feature (ips isolation or a firewall) to block ssh.
| improve this answer | |
  • I don't think the Pi has firewall, ufw isn't installed and I don't think it comes with a pre-installed firewall, and I didn't installed any personally. When it comes to the router, it's CISCO EPC3925 and I think it doesn't have such features. After all, my other Pi works, and even this one worked from one laptop up to some point, now it doesn't (I didn't change anything). Really weird. I'll have to reinstall entire operating system on the Pi and try again but would prefer not do to that, hopefully someone knows what's wrong and can troubleshoot the issue. – adder Dec 17 '19 at 16:26
  • "sudo iptables-save" ? "sudo netstat --tcp --listening --programs --numeric" ? – Yurko Dec 17 '19 at 17:09
  • where do I run those commands, on the Pi? – adder Dec 17 '19 at 17:20
  • Yes, on PI box. As a user starting with "sudo" or as root without "sudo". – Yurko Dec 17 '19 at 17:22
  • Alright, see updated question for the output. – adder Dec 17 '19 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.