There was a HW issue that rebooted several virtual machines and one of them had the following problem: from that point the server isn't reachable via network (ssh) because it gives error message:

Connection refused

We tried to log in via console, but if we type any username in, it gives:

Login incorrect

Error message. Interesting thing that sshd_config has "UsePam No" -> maybe it is not a PAM problem?

We can only enter single user mode with replacing the "ro" to "rw init=/sysroot/bin/sh" at the line in linux16, CTRL+X, then chroot /sysroot.

Question: What should we check when we are in single user mode?

We already restored the "system-auth-ac" and "password-auth-ac", the same error message after reboot.

Another interesting thing is that when we are in single user mode, after chroot, the "/var/log" doesn't exists. There is no lvs command in chroot, but if we exit from chroot, and type lvs, there is only the root LV. Where are the other LVs?

  • Booting off the dvd into rescue mode is probably most useful, might be why you're not getting all the filesystems. I'd also pay attention to the boot to see if you're getting a filesystem error that's halting the boot process. – Bratchley Mar 6 '17 at 16:24
  • yes, of course, but this is a restricted environment and we will not get a DVD to boot from it in rescue mode.. sadly.. so we can only try single user mode – Marina Ala Mar 6 '17 at 16:26
  • Try "rw single" kernel command line instead of "rw init=...sh". This should mount disks (including the /var/log) before giving you the shell. – VPfB Mar 6 '17 at 16:29
  • @VPfB: thanks, that solved the issue, posting it as answer – Marina Ala Mar 7 '17 at 13:27

Replace the "ro" to "rw single" at the line in linux16, CTRL+X, then chroot /sysroot to have a proper single mode, /var/log is even reachable then.

Check the /var/log/messages for any error messages.

We found out that there are error messages that points to that /var is full (even the 5% reserved space too).

We increased the /var and rebooted, login works again!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.