1

I have a desktop computer that has Ubuntu 16.04 Server running on it. It's in my living room, connected to the router via ethernet cable. IP address is static (192.168.50.249) and DHCP on the router is configured to stop at 240. I have a bunch of services running there: PG 10.3, SSHd, Docker and some other services. I use it for my development: I connect to the server to use PG and some apps that I have running in docker. I'm using a MBP for work and connect from it to this server via WiFi.

After some time (it's unpredictable from minutes to days), the application-level network connections stop working. I can still ping the server and telnet to any port, but I cannot actually use any services. For psql it says "Is server actually listening and listening on port 5432" for SSH it returns 'connection refused for 192.168.50.249, port 22'.

If I telnet to port 22, it connects but then quits with "protocol mismatch" message after first key press.

The only thing that helps is hard reset button on the server.

If I look into /var/log/auth I don't see any traces of attempted connections. Successful connections are logged fine there. I've tried disabling password connection (connect using my RSA key) and enable it back but it didn't help. Looks like it's not reaching SSH service at all. I'm not an admin, just a regular linux user, and I have exhausted my modest list of things to try. I've also tried checking if firewall is running (but it's not):

 $ sudo service ufw status

● ufw.service - Uncomplicated firewall
   Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2018-03-20 13:11:26 CDT; 3h 38min ago
  Process: 470 ExecStart=/lib/ufw/ufw-init start quiet (code=exited, status=0/SUCCESS)
 Main PID: 470 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/ufw.service


 $ sudo ufw status
Status: inactive

I've also looked through /var/log/syslog but didn't find anything relevant there.

Please point me into the direction of what to try.

Also here is the output of lspci:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1450
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 1451
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a804
02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43bb (rev 02)
02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b7 (rev 02)
02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b2 (rev 02)
03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
09:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
0a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457

and uname

$ uname -a Linux gladstone 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

UPDATE:

It happened again today and I ran nmap:

sudo nmap -sS -A 192.168.50.249
Password:
Starting Nmap 7.70 ( https://nmap.org ) at 2018-04-04 07:55 CDT
Nmap scan report for gladstone (192.168.50.249)
Host is up (0.0021s latency).
Not shown: 999 closed ports
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 7.2p2 Ubuntu 4ubuntu2.4 (Ubuntu Linux; protocol 2.0)
MAC Address: 88:D7:F6:C4:F0:2B (Asustek Computer)
Device type: general purpose
Running: Linux 3.X|4.X
OS CPE: cpe:/o:linux:linux_kernel:3 cpe:/o:linux:linux_kernel:4
OS details: Linux 3.2 - 4.9
Network Distance: 1 hop
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

TRACEROUTE
HOP RTT     ADDRESS
1   2.09 ms gladstone (192.168.50.249)

So it's the same computer, it's accessible via telnet, but SSH is not working. I'm stuck.

Telnet works:

telnet 192.168.50.249 22
Trying 192.168.50.249...
Connected to gladstone.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
d
Protocol mismatch.
Connection closed by foreign host.

SSH not working:

ssh -vvv 192.168.50.249
OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 192.168.50.249 port 22.
debug1: Connection established.
debug1: identity file /Users/valentin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to 192.168.50.249:22 as 'valentin'
debug3: hostkeys_foreach: reading file "/Users/valentin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/valentin/.ssh/known_hosts:37
debug3: load_hostkeys: loaded 1 keys from 192.168.50.249
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
Connection closed by 192.168.50.249 port 22

However after hardware reset:

ssh -vvv 192.168.50.249
OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 192.168.50.249 port 22.
debug1: Connection established.
debug1: identity file /Users/valentin/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/valentin/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to 192.168.50.249:22 as 'valentin'
debug3: hostkeys_foreach: reading file "/Users/valentin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/valentin/.ssh/known_hosts:37
debug3: load_hostkeys: loaded 1 keys from 192.168.50.249
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:L1h3Nhz+wvvg7SyJGUyNmMMSTPSXQ6URWv2xuCei3IU
debug3: hostkeys_foreach: reading file "/Users/valentin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/valentin/.ssh/known_hosts:37
debug3: load_hostkeys: loaded 1 keys from 192.168.50.249
debug1: Host '192.168.50.249' is known and matches the ECDSA host key.
debug1: Found key in /Users/valentin/.ssh/known_hosts:37
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /Users/valentin/.ssh/id_rsa (0x7f9cafc02be0)
debug2: key: /Users/valentin/.ssh/id_dsa (0x0)
debug2: key: /Users/valentin/.ssh/id_ecdsa (0x0)
debug2: key: /Users/valentin/.ssh/id_ed25519 (0x0)
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:ZtUc3syi/EohVjwVTuqqvAYaLaDO58+yE6xXJ5iTOeg /Users/valentin/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 535
debug2: input_userauth_pk_ok: fp SHA256:ZtUc3syi/EohVjwVTuqqvAYaLaDO58+yE6xXJ5iTOeg
debug3: sign_and_send_pubkey: RSA SHA256:ZtUc3syi/EohVjwVTuqqvAYaLaDO58+yE6xXJ5iTOeg
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.50.249 ([192.168.50.249]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 5 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TMPDIR
debug3: Ignored env XPC_FLAGS
debug3: Ignored env Apple_PubSub_Socket_Render
debug3: Ignored env TERM
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env SECURITYSESSIONID
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env SHELL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env USER
debug3: Ignored env PATH
debug3: Ignored env SHLVL
debug3: Ignored env PWD
debug3: Ignored env OLDPWD
debug3: Ignored env ZSH
debug3: Ignored env PAGER
debug3: Ignored env LESS
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env LSCOLORS
debug3: Ignored env RBENV_SHELL
debug3: Ignored env _
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-116-generic x86_64)

So the difference is:

debug1: SSH2_MSG_KEXINIT sent <<-- this is where it stops
debug3: receive packet: type 20  <<-- this packet is never received when broken
debug1: SSH2_MSG_KEXINIT received
  • Your router may be causing the problem. Are there any network debugging tools and/or logs on the router? During "failure" mode connecting to the server, are there other network hosts which can be reached through ssh? It's possible that the arp table in the router is getting corrupted over time... does the problem clear when only the router is reset? or does the server need to be rebooted as well as the router? – RubberStamp Mar 20 '18 at 22:27
  • @RubberStamp only server needs to be rebooted. I don't think it's a router, because the moment I reboot the server it becomes accessible instantly – Valentin Mar 20 '18 at 23:56
  • Something else could be using that IP address. Try using nmap next time the network services fail - nmap -sS -A 192.168.50.249. That will give you the hostname and some info on the services. Is the hostname the same? Is the SSH key fingerprint the same? – dogoncouch Mar 21 '18 at 2:16
  • @dogoncouch I did this experiment: I launched ping on my laptop (it started pinging the server), then unplugged the Ethernet cable (pinging started to time out), plugged it back in, pinging resumed. This led me to think it's not a IP collision. – Valentin Mar 21 '18 at 14:05
  • @Valentin You could try a tcpdump on the interface, and see if your SSH packets are getting through. I can't think of anything else at the moment. – dogoncouch Mar 21 '18 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.