16

I have a problem which is reproducible on Linux Ubuntu VMs (14.04 LTS) created in Azure.

After installing systemd package through script, the system refuses new ssh connections, infinitely.

System is booting up.

Connection closed by xxx.xxx.xxx.xxx

The active ssh connection is maintained though. There is no /etc/nologin file present in the system.

The only option I see is a hard reset which solves the problem. But how do I avoid it?

Here is the script I am using:

#!/bin/bash

# Script input arguments
user=$1
server=$2

# Tell the shell to quote your variables to be eval-safe!

printf -v user_q '%q' "$user"
printf -v server_q '%q' "$server"
#

SECONDS=0
address="$user_q"@"$server_q"

function run {
    ssh "$address" /bin/bash "$@"
}

run << SSHCONNECTION
    # Enable autostartup

        # systemd is required for the autostartup
        sudo dpkg-query -W -f='${Status}' systemd 2>/dev/null | grep -c "ok installed" > /home/$user_q/systemd-check.txt
        systemdInstalled=\$(cat /home/$user_q/systemd-check.txt)
                
        if [[ \$systemdInstalled -eq 0 ]]; then
            echo "Systemd is not currently installed. Installing..."
            
            # install systemd
            sudo apt-get update
            sudo apt-get -y install systemd

        else
            echo "systemd is already installed. Skipping this step."
        fi

SSHCONNECTION
2
  • Is the system hanging, or is it simply not starting the secure shell daemon? The question states one; the body of the post implies that it could well be the other.
    – DopeGhoti
    Commented Jan 5, 2017 at 20:26
  • @DopeGhoti There is no way for me to check what's going on since I cannot connect to the machine remotely. I will update the question to make it clearer.
    – Alex
    Commented Jan 5, 2017 at 20:27

5 Answers 5

27

I suspect there is a /etc/nologin file (whose content would be "System is booting up.") that is not removed after the systemd installation.

[update] What affects you is a bug that was reported on Ubuntu's BTS last December. It is due to a /var/run/nologin file (= /run/nologin since /var/run is a symlink to /run) that is not removed at the end of the systemd installation.

/etc/nologin is the standard nologin file. /var/run/nologin is an alternate file that may be used by the nologin PAM module (man pam_nologin).

Note that none of the nologin files affect connections by user root, only regular users are prevented from logging in.

3
  • I reproduced the issue, there is no /etc/nologin file present. Active SSH session is maintained, however new ones are refused until I restart the machine.
    – Alex
    Commented Jan 5, 2017 at 22:41
  • I also checked /etc/shadow and the account is not locked
    – Alex
    Commented Jan 5, 2017 at 22:44
  • 1
    @Alex Answer updated.
    – xhienne
    Commented Jan 6, 2017 at 8:15
12

@xhienne gave me the right direction.

After searching through the file system I found /run/nologin (@xhienne suggested /etc/nologin) file, removing which solved the problem.

The condition existed in /usr/lib/tmpfiles.d/systemd.conf

I will include this step in my script.

sudo rm /run/nologin
1
  • Glad it works. I have updated my answer.
    – xhienne
    Commented Jan 6, 2017 at 8:14
2
Note:  This answer is applicable whether or not systemd was recently installed or not.
       The issue was observed even after systemd had been installed a long time.

The Mageia distribution bug tracker seems to have a related issue open: Bug 21080 - ssh login disabled by /run/nologin after a reboot .

After experiencing this problem quite frequently, finding the tracker helped identify a workaround that could be more appropriate than simply removing the /run/login file.

Here is some data related to queries for information in that bug tracker:

$ ls -l /run/nologin 
-rw-r--r-- 1 root root 42 Mar  6 10:11 /run/nologin
$ cat /run/nologin
"System is booting up. See pam_nologin(8)"
$ date
Tue Mar  6 11:10:38 CST 2018
$ uptime
11:15:10 up  1:04,  0 users,  load average: 0.07, 0.07, 0.08
$ systemctl status systemd-user-sessions.service
● systemd-user-sessions.service - Permit User Sessions
   Loaded: loaded (/usr/lib/systemd/system/systemd-user-sessions.service; static
   Active: inactive (dead)
     Docs: man:systemd-user-sessions.service(8)
$ systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After  systemd-user-sessions.service --no-pager
Requires=system.slice sysinit.target
Requisite=
Wants=
BindsTo=
PartOf=
[email protected] prefdm.service crond.service multi-user.target plymouth-quit-wait.service session-c2.scope display-manager-failure.service systemd-ask-password-wall.service session-c1.scope [email protected] shutdown.target [email protected] user-983.slice user-1000.slice plymouth-quit.service
After=system.slice systemd-journald.socket remote-fs.target network.target systemd-journal-flush.service sysinit.target nss-user-lookup.target basic.target

The bug tracker and the information above seems to show that the problem is actually due to a failure to start the systemd-user-sessions.service daemon.

This is in fact what happens in my case, so the following workaround temporarily corrects the banned login condition:

$ sudo systemctl start systemd-user-sessions.service

After doing this, the /run/nologin file is no longer present, and, one can SSH from another system. Note, however, that this is not reliable as sometimes the user does not have access to the console of the affected system.

1
  • I got it to run automatically by editing, /lib/systemd/system/ssh.service, the [Service] section, adding the line ExecStartPost=systemctl start systemd-user-sessions.service But I am running WSL2 version of Ubuntu 20.04LTS, and using this I cannot get it to automatically start my SSH Server service. But you've put me 50% closer to being able to SSH into WSL2 for debugging. Thanks.
    – PfunnyGuy
    Commented Jun 2, 2022 at 20:43
0

I had this exact same problem but I think several scenarios can create it.

In my case, to enable remote access again I had to request KVM for direct access to our remote server and then:

# 1. Start SSH service
/etc/init.d/ssh start

# 2. Remove the nologin file
rm /run/nologin

But at the KVM screen I could actually see that it booted into emergency mode!

Previously, I had been making some disk/partition changes (increasing inodes) which generated a new UUID and forgot to add it the /etc/fstab file.

After issuing the command:

blkid

...and copy pasting the new UUID on the fstab file, I was able reboot the server again with no issues and remote SSH access was fine after that.

0

In the /etc/ssh/sshd_config set UsePAM to no

UsePAM no
2
  • 2
    What would this do and what would the consequences be?
    – Kusalananda
    Commented Jan 1, 2020 at 12:49
  • This answer doesn't seem to apply to this situation-- it doesn't explain why the user sees the "System is booting up" text or explain how installing systemd generated the broken configuration.
    – Jeff Schaller
    Commented Jan 1, 2020 at 13:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .