3

I use a VM with ssh and X forwarding to run certain applications. When launching the VirtualBox VM, it takes several minutes for ssh to come online, even though the machine is at the desktop environment's login prompt in no time. The machine does not use a lot of resources (CPU, network) while sshd is starting.

Other things that might need to understand the issue:

  • The host is a Debian Buster (testing) system running OpenSSH_7.9p1 Debian-1, OpenSSL 1.1.1 11 Sep 2018
  • When I log in on the desktop environment, it also takes a bit, but usually not multiple minutes. It accepts my login (or tells me I typo'd) immediately and switches to a blank screen, but takes a while to show the actual desktop.
  • Once the desktop is shown and I can start to debug the issue, the issue has disappeared: sshd invariably works when the desktop environment has loaded. They both seem to block on the same thing.
  • While sshd is not yet up, the system responds with "connection refused", i.e. it can reach the host and port, but sshd seems not to have bound to it.
  • I can also ping the host, immediately from the moment the boot screen changes to the desktop environment's login prompt. The system seems fully ready, except sshd does not start (or bind to the interface).

Switching to a virtual terminal does not work because it switches to a virtual terminal of my host system, even if set the keyboard to be captured. I could prevent the desktop env from booting and get a terminal immediately, but I only have a short amount of time (2-3 minutes I'd say) so I would have to reboot after every few commands or know where to look in order to figure it out.

How could I debug this issue effectively? Is there a likely cause that I could check?

6
  • 1
    maybe dns resolve error. Commented Nov 22, 2018 at 9:29
  • @IporSircer I was thinking DNS might be it, but what could I do to fix it? Resolvconf seems to work correctly as, after booting, dns works fine.
    – Luc
    Commented Nov 22, 2018 at 9:50
  • It seems that I have the same problem: Debian in a VM, takes more than a minutes to start sshd. I tried tuning it via /etc/ssh/sshd_conf, disabling DNS lookup, added verbose logs, etc. but it is still slow and I am not sure what is causing it. It looks like sshd starts to print logs (in /var/log/auth.log) only after it has been fully started, so it does not help much in decided why it is starting so slow. Commented Dec 11, 2018 at 10:26
  • The interesting thing is that when I login into that Debian VM (i.e. type my login and password in the terminal), then it start quicker: "Startup finished in 2.878s (kernel) + 9.890s (userspace)" (in /var/log/daemon.log). But if I just start the VM without logging in and want to log in via SSH from my main machine, sshd starts for more than a minute (I can see it in /var/log/daemon.log again). It seems like logging in causes it to start quicker (perhaps some other service, like networking starts as soon as one logs in, I don't know) Commented Dec 11, 2018 at 10:32
  • @dying_sphynx That sounds 100% like what I'm seeing. I haven't had a chance to debug this issue yet, maybe next week. I'll keep you posted if I do!
    – Luc
    Commented Dec 11, 2018 at 10:51

1 Answer 1

6

Most likely lack of entropy. In the comments, you have confirmed that doing some activity on the box makes sshd start quicker - that happens because your activity contributes to the entropy sources.

Install (and make sure it gets started) haveged (or, in older distributions, rngd).

6
  • I hadn't thought of that as a possible cause, and that would indeed match the symptoms! As far as I know, though, Debian saves an entropy file on shutdown. I'll investigate at work next week. (Though the issue seems to have resolved itself at some point after installing updates, I still have an old VM to test with. If this was the issue, I'll mark the answer as accepted of course.)
    – Luc
    Commented Apr 13, 2019 at 19:07
  • It does not appear to be a lack of entropy after all. I installed haveged and wrote a little shell script that runs on startup and logs available entropy. At startup the entropy is like 72±10 bits and within half a second (at least with haveged installed) it jumps to >2400 bits. When I try to log in 15 minutes after booting the VM, it still took 15 seconds for ssh to log in (a more typical login time is <1 second, since it's localhost and a modern laptop).
    – Luc
    Commented Nov 19, 2019 at 8:50
  • Logging in is not sshd starting up, though. If your login process is delayed, it could be the reverse DNS timing out, which can be (mostly) disabled with UseDNS no in the sshd config. In some cases it has been reported that GSSAPIAuthentication no can help as well.
    – Richlv
    Commented Nov 20, 2019 at 8:47
  • Setting both of those sadly doesn't seem to help. Not sure if it's rdns: I would expect that remains slow because it would retry the query, but the issue only occurs after booting and is then fine. I also can't always reproduce it: rebooting a VM is often fine, but typically when I start it fresh in the morning, it's not (I'm wondering if maybe suspending the host system is what resets it, but it's also not as if other VMs are fine after the first one took its sweet time, so it's not just a host thing).
    – Luc
    Commented Nov 21, 2019 at 12:05
  • 1
    thanks! Installing haveged fixed the problem.
    – Alek
    Commented Feb 6, 2020 at 19:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .