ssh.service will not start at boot if using ListenAddress in sshd_config but it will if started manuall after boot [duplicate]

Question

I have ListenAddress in my /etc/ssh/sshd_config. ssh.service will not start at boot but it will start if I run service ssh start after boot.

Here is the error from /var/log/syslog:

$ sudo cat /var/log/syslog | grep -i ssh
Feb  9 10:03:50 nook systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Feb  9 10:03:50 nook systemd[1]: ssh.service: Unit entered failed state.
Feb  9 10:03:50 nook systemd[1]: ssh.service: Failed with result 'exit-code'.

I've looked online but can't seem to figure out what is wrong. It works if I start it manually after boot but it won't start at boot. :/

This is my /etc/ssh/sshd_config:

UsePAM yes
PrintMotd no
AcceptEnv LANG LC_*
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
KexAlgorithms [email protected],ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
MACs [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]
LogLevel VERBOSE
Subsystem sftp  /usr/lib/ssh/sftp-server -f AUTHPRIV -l INFO
UsePrivilegeSeparation sandbox
Protocol 2
X11Forwarding no
AllowTcpForwarding no
AllowStreamLocalForwarding no
GatewayPorts no
PermitTunnel no
PermitEmptyPasswords no
IgnoreRhosts yes
UseDNS yes
Compression no
TCPKeepAlive no
AllowAgentForwarding no
PermitRootLogin no
AllowGroups sshusers
ClientAliveCountMax 0
ClientAliveInterval 300
LoginGraceTime 30
ListenAddress 192.168.1.100   # IP of server
MaxAuthTries 2
MaxSessions 2
MaxStartups 2
ChallengeResponseAuthentication yes

My full complete boot log from /var/log/syslog is at https://pastebin.com/SfjPQzXu.

Answer 1

I had the same problem. The root cause of the issue is that out of the box sshd unit file has just network.target as a dependency. When you specify IP address explicitly in sshd config file, it must be configured on an interface prior to sshd start, what network.target is not supposed to take care of. To resolve the issue you need to do in console under root:

# systemctl edit sshd

and after editor opens, define a dependency on network-online.target:

[Unit]
After=network-online.target

'systemctl edit ...' command creates a file

/etc/systemd/system/${SERVICE_NAME}.d/override.conf

with your input and reloads dependency tree after saving, which you usually achieve doing "systemctl daemon-reload" after manual editing unit files.

Please note: do not edit unit files under /lib/systemd/system as they are manged by your package manager and will be overwritten on next update.

However putting network-online.target as a unit dependency is still not enough. On most of Linux based operating systems shipped with systemd, network-online.target starts immediately after network.target. This is because another service which network-online.target depends on is disabled by default. It is called systemd-networkd-wait-online.service. So make sure systemd-networkd-wait-online.service is enabled

# systemctl is-enabled systemd-networkd-wait-online
disabled
# systemctl enable systemd-networkd-wait-online

UPDATE: The advice works only for operating systems using systemd-networkd for network management. For systems without systemd-networkd enabled, "Restart=on-failure" option might help along with "RestartSec=5" to control sleep between start up attempts