5

SOLVED! The machine is a Dell Poweredge system which ran what they call iDRAC in the background - iDRAC in its turn ran a SSH-server which conflicted with the one I installed in Debian. The solution for me was just to disable iDRAC since I don't use it, and I'm glad I don't. I have no idea how that conflict should've been solved.

Just installed Debian Stretch on a machine. Minimal install haven't done pretty much anything but installing openssh-server and setting up my network interface. The problem I have seems to be related to the networking service.

Straight from boot I'm not able to connect to my server via SSH remotely. The server has internet connection at this time, I'm able to ping it and ping from it. At connection it starts out good but then a popup asks for a password for the user, typing in the password doesn't work - should already have been sent by the client anyway. When I run systemctl restart networking.service on the server SSH suddenly starts working.

openssh-server is running with default configurations.

My SSH client(Bitvise) log says:

15:08:26.493 Started a new SSH2 session.
15:08:26.493 Connecting to SSH2 server 171.xxx.xxx.xxx:22.
15:08:26.493 Connection established.
15:08:26.681 Server version: SSH-2.0-OpenSSH_5.8 <----
15:08:26.681 First key exchange started.
15:08:27.289 Received a new host key from the server. Algorithm: xxxxxxxxxxx
15:08:27.320 First key exchange completed using ecdh-sha2/nistp521. xxxxxxxxxxx
15:08:27.320 Attempting password authentication.
15:08:28.724 Authentication failed. Remaining authentication methods: 'password'.
15:08:30.581 Authentication aborted on user's request.
15:08:30.596 The SSH2 session has been terminated.

Note the server version SSH-2.0-OpenSSH_5.8

telnet 174.xxx.xxx.xxx 22

responds with SSH-2.0-OpenSSH_5.8

On the server SSH seems to be running just fine:

systemctl status sshd.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2017-06-28 11:16:52 CEST; 4h 10min ago
Main PID: 598 (sshd)
Tasks: 1 (limit: 7372)
CGroup: /system.slice/ssh.service
       └─ 598 /usr/sbin/sshd -D

systemd[1]: Starting OpenBSD Secure Shell server...
sshd[598]: Server listening on 0.0.0.0 port 22.
sshd[598]: Server listening on :: port 22.
systemd[1]: Started OpenBSD Secure Shell server.

Status for networking.service shows this

systemctl status networking.service
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2017-06-28 11:16:49 CEST; 4h 10min ago
  Docs: man:interfaces(5)
Process: 471 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
Process: 335 ExecStartPre=/binsh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited, status=0/SUCCESS)
Main PID: 1120 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 7372)
CGroup: /system.slice/networking.service

systemd[1]: Starting Raise network interfaces...
systemd[1]: Started Raise network interfaces.

After restarting the network service

systemctl restart networking.service

After running this command SSH all of a sudden starts working.

SSH client log says:

15:08:57.179 Started a new SSH2 session.
15:08:57.179 Connecting to SSH2 server 171.xxx.xxx.xxx:22.
15:09:00.205 Connection established.
15:09:00.205 Server version: SSH-2.0-OpenSSH_7.4p1 Debian-10 <----
15:09:00.205 First key exchange started.
15:09:00.283 Received a new host key from the server. Algorithm: xxxxxxxxxxx
15:09:00.314 First key exchange completed using ecdh-sha2/nistp521. xxxxxxxxxxx
15:09:00.314 Attempting password authentication.
15:09:00.330 Authentication completed.
15:09:00.470 Terminal channel opened.

The odd thing here is that now it shows server version SSH-2.0-OpenSSH_7.4p1 Debian-10

telnet 174.xxx.xxx.xxx 22

responds with SSH-2.0-OpenSSH_7.4p1 Debian-10

systemctl status sshd.service shows same as before

Status for networking.service gives shows this

systemctl status networking.service
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2017-06-28 15:12:49 CEST; 1min 12s ago
  Docs: man:interfaces(5)
Main PID: 1292 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 7372)
CGroup: /system.slice/networking.service

systemd[1]: Starting Raise network interfaces...
systemd[1]: Started Raise network interfaces.

/etc/network/interfaces

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
 iface lo inet loopback

auto eno1
 iface eno1 inet static
 address 174.xxx.xxx.29
 netmask 255.255.255.248
 gateway 174.xxx.xxx.25
 broadcast 174.xxx.xxx.31
 dns-nameservers 8.8.8.8 8.8.4.4

/etc/hosts

127.0.0.1 localhost.localdomain localhost
127.0.1.1 boris.secnet.sec boris

/etc/hostname

boris

/etc/resolv.conf

nameserver 8.8.8.8
nameserver 8.8.4.4

Made a resolv.conf and added nameservers there. Had issues with just having them in /etc/network/interfaces. Tried both with and without the resolv.conf and specifying them in /etc/network/interfaces.

NetworkManager enabled/disabled, no difference in behaviour. No firewall. The server is connected straight to the internet, same goes for my client machine.

So, what happens with SSH when systemctl restart network.services is invoked? Why does it throw back a different SSH server versions before and after the networking service is restarted?

Why won't the networking service do, whatever it does when restarting it manually, at boot?

I'm completely lost on this one, been testing for days. Worked perfectly fine with Debian Wheezy on the same net and interface settings before I upgraded. Made a full reinstall, formatted disk and installed Stretch.

Does anyone have a clue about what the problem might be? Or how to troubleshoot.

  • It looks weird indeed. What I'd do is I'd first disable and then reenable the ssh service and see if nothing crops up in relation to service configuration. – Tomasz Jun 28 '17 at 18:18
  • 1
    Possible hint: in your first Bitvise log the IP is 174.xxx.xxx.xxx, while in the second one it's 171.xxx.xxx.xxx. – Ferenc Wágner Jun 30 '17 at 6:58
  • @FerencWágner Thanks for your reply! It's not that, the IPs are the same. I just masked them here. – Henry.t.t Jun 30 '17 at 12:26
  • @sourcejedi Thanks for your reply. I've solved the problem now. It was indeed a problem with "Lights out management". The server ran a SSH-server in the background. – Henry.t.t Jul 5 '17 at 15:34
  • @Henry.t.t Thanks for coming back with your answer! Editing the question to include the answer doesn't really fit the system here. You're able, and actively encouraged to, write an answer even if it's on one of your own questions, and mark it accepted. I didn't notice your edit initially, so I converted my comment into an answer. – sourcejedi Jul 5 '17 at 16:30
1

SOLVED! The machine is a Dell Poweredge system which ran what they call iDRAC in the background - iDRAC in its turn ran a SSH-server which conflicted with the one I installed in Debian. The solution for me was just to disable iDRAC since I don't use it, and I'm glad I don't. I have no idea how that conflict should've been solved.

  • How did iDRAC get on that server? Did they provide it with the OS and iDRAC installed? Did you install it? Is there an update mechanism for it? I'm curious because I would be hesitant to run some proprietary management tool on my servers. Said tool listening on the network makes me even more paranoid; my experience with the quality and security updates of such tools has been... Less than stellar. – marcelm Oct 24 '18 at 21:12
0

The version mismatch seems very suspicious to me.

systemctl status is pretty unambiguous, that you're successfully running /usr/sbin/sshd. It would be very strange if the system initially booted with a non-Debian binary in /usr/sbin/sshd, but once it was booted, the proper Debian binary was available at /usr/sbin/sshd.

Since you insist you're connecting to the same IP address in both cases :-P, something must be intercepting the connections before they reach your SSH daemon. I think it's likely this is some very confused "Lights out management" on the server, intercepting the ssh connection at a hardware level. (For example, the failed SSH connection attempts wouldn't show up in Debian's /var/log/auth.log, or even in tcpdump port ssh).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.