4

There was a HW issue that rebooted several virtual machines and one of them had the following problem: from that point the server isn't reachable via network (ssh) because it gives error message:

Connection refused

We tried to log in via console, but if we type any username in, it gives:

Login incorrect

Error message. Interesting thing that sshd_config has "UsePam No" -> maybe it is not a PAM problem?

We can only enter single user mode with replacing the "ro" to "rw init=/sysroot/bin/sh" at the line in linux16, CTRL+X, then chroot /sysroot.

Question: What should we check when we are in single user mode?

We already restored the "system-auth-ac" and "password-auth-ac", the same error message after reboot.

Another interesting thing is that when we are in single user mode, after chroot, the "/var/log" doesn't exists. There is no lvs command in chroot, but if we exit from chroot, and type lvs, there is only the root LV. Where are the other LVs?

4
  • Booting off the dvd into rescue mode is probably most useful, might be why you're not getting all the filesystems. I'd also pay attention to the boot to see if you're getting a filesystem error that's halting the boot process.
    – Bratchley
    Mar 6, 2017 at 16:24
  • yes, of course, but this is a restricted environment and we will not get a DVD to boot from it in rescue mode.. sadly.. so we can only try single user mode
    – Marina Ala
    Mar 6, 2017 at 16:26
  • Try "rw single" kernel command line instead of "rw init=...sh". This should mount disks (including the /var/log) before giving you the shell.
    – VPfB
    Mar 6, 2017 at 16:29
  • @VPfB: thanks, that solved the issue, posting it as answer
    – Marina Ala
    Mar 7, 2017 at 13:27

2 Answers 2

4

Replace the "ro" to "rw single" at the line in linux16, CTRL+X, then chroot /sysroot to have a proper single mode, /var/log is even reachable then.

Check the /var/log/messages for any error messages.

We found out that there are error messages that points to that /var is full (even the 5% reserved space too).

We increased the /var and rebooted, login works again!

0

The issue could likely be because networking wasn't selected on install tailling /var/log/secure while attempting to login

PAM unable to dlopen(/usr/lib64/security/pam_pkcs11.so): /usr/lib64/security/pam_pkcs11.so: cannot open shared object file: No such file or directory
<83>2020-12-16T09:18:08.672843-06:00 tempbuild19 login: PAM adding faulty module: /usr/lib64/security/pam_pkcs11.so

to resolve, install the package that has the .so file

yum install pam_pkcs11

reference: https://forums.centos.org/viewtopic.php?t=61314

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .