9

I have a problem which is reproducible on Linux Ubuntu VMs (14.04 LTS) created in Azure.

After installing systemd package through script, the system refuses new ssh connections, infinitely.

System is booting up.

Connection closed by xxx.xxx.xxx.xxx

The active ssh connection is maintained though. There is no /etc/nologin file present in the system.

The only option I see is a hard reset which solves the problem. But how do I avoid it?

Here is the script I am using:

#!/bin/bash

# Script input arguments
user=$1
server=$2

# Tell the shell to quote your variables to be eval-safe!

printf -v user_q '%q' "$user"
printf -v server_q '%q' "$server"
#

SECONDS=0
address="$user_q"@"$server_q"

function run {
    ssh "$address" /bin/bash "$@"
}

run << SSHCONNECTION
    # Enable autostartup

        # systemd is required for the autostartup
        sudo dpkg-query -W -f='${Status}' systemd 2>/dev/null | grep -c "ok installed" > /home/$user_q/systemd-check.txt
        systemdInstalled=\$(cat /home/$user_q/systemd-check.txt)

        if [[ \$systemdInstalled -eq 0 ]]; then
            echo "Systemd is not currently installed. Installing..."

            # install systemd
            sudo apt-get update
            sudo apt-get -y install systemd

        else
            echo "systemd is already installed. Skipping this step."
        fi

SSHCONNECTION
  • Is the system hanging, or is it simply not starting the secure shell daemon? The question states one; the body of the post implies that it could well be the other. – DopeGhoti Jan 5 '17 at 20:26
  • @DopeGhoti There is no way for me to check what's going on since I cannot connect to the machine remotely. I will update the question to make it clearer. – Alex Jan 5 '17 at 20:27
11

I suspect there is a /etc/nologin file (whose content would be "System is booting up.") that is not removed after the systemd installation.

[update] What affects you is a bug that was reported on Ubuntu's BTS last December. It is due to a /var/run/nologin file (= /run/nologin since /var/run is a symlink to /run) that is not removed at the end of the systemd installation.

/etc/nologin is the standard nologin file. /var/run/nologin is an alternate file that may be used by the nologin PAM module (man pam_nologin).

Note that none of the nologin files affect connections by user root, only regular users are prevented from logging in.

  • I reproduced the issue, there is no /etc/nologin file present. Active SSH session is maintained, however new ones are refused until I restart the machine. – Alex Jan 5 '17 at 22:41
  • I also checked /etc/shadow and the account is not locked – Alex Jan 5 '17 at 22:44
  • @Alex Answer updated. – xhienne Jan 6 '17 at 8:15
10

@xhienne gave me the right direction.

After searching through the file system I found /run/nologin (@xhienne suggested /etc/nologin) file, removing which solved the problem.

The condition existed in /usr/lib/tmpfiles.d/systemd.conf

I will include this step in my script.

sudo rm /run/nologin
  • Glad it works. I have updated my answer. – xhienne Jan 6 '17 at 8:14
2
Note:  This answer is applicable whether or not systemd was recently installed or not.
       The issue was observed even after systemd had been installed a long time.

The Mageia distribution bug tracker seems to have a related issue open: Bug 21080 - ssh login disabled by /run/nologin after a reboot .

After experiencing this problem quite frequently, finding the tracker helped identify a workaround that could be more appropriate than simply removing the /run/login file.

Here is some data related to queries for information in that bug tracker:

$ ls -l /run/nologin 
-rw-r--r-- 1 root root 42 Mar  6 10:11 /run/nologin
$ cat /run/nologin
"System is booting up. See pam_nologin(8)"
$ date
Tue Mar  6 11:10:38 CST 2018
$ uptime
11:15:10 up  1:04,  0 users,  load average: 0.07, 0.07, 0.08
$ systemctl status systemd-user-sessions.service
● systemd-user-sessions.service - Permit User Sessions
   Loaded: loaded (/usr/lib/systemd/system/systemd-user-sessions.service; static
   Active: inactive (dead)
     Docs: man:systemd-user-sessions.service(8)
$ systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After  systemd-user-sessions.service --no-pager
Requires=system.slice sysinit.target
Requisite=
Wants=
BindsTo=
PartOf=
Before=getty@tty1.service prefdm.service crond.service multi-user.target plymouth-quit-wait.service session-c2.scope display-manager-failure.service systemd-ask-password-wall.service session-c1.scope user@983.service shutdown.target user@1000.service user-983.slice user-1000.slice plymouth-quit.service
After=system.slice systemd-journald.socket remote-fs.target network.target systemd-journal-flush.service sysinit.target nss-user-lookup.target basic.target

The bug tracker and the information above seems to show that the problem is actually due to a failure to start the systemd-user-sessions.service daemon.

This is in fact what happens in my case, so the following workaround temporarily corrects the banned login condition:

$ sudo systemctl start systemd-user-sessions.service

After doing this, the /run/nologin file is no longer present, and, one can SSH from another system. Note, however, that this is not reliable as sometimes the user does not have access to the console of the affected system.

0

I had this exact same problem but I think several scenarios can create it.

In my case, to enable remote access again I had to request KVM for direct access to our remote server and then:

# 1. Start SSH service
/etc/init.d/ssh start

# 2. Remove the nologin file
rm /run/nologin

But at the KVM screen I could actually see that it booted into emergency mode!

Previously, I had been making some disk/partition changes (increasing inodes) which generated a new UUID and forgot to add it the /etc/fstab file.

After issuing the command:

blkid

...and copy pasting the new UUID on the fstab file, I was able reboot the server again with no issues and remote SSH access was fine after that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.