0

My cron jobs have stopped working on my CentOS 7 server. The server is running WHM/cPanel.

It seems like it is an issue with PAM service because in /var/log/secure I can see the following errors when the cron jobs try to run:

Jun 24 10:45:01 server1 crond[22400]: pam_access(crond:account): auth could not identify password for [root]
Jun 24 10:45:01 server1 crond[22404]: pam_access(crond:account): auth could not identify password for [admin]
Jun 24 10:45:01 server1 crond[22400]: pam_localuser(crond:account): auth could not identify password for [root]
Jun 24 10:45:01 server1 crond[22402]: pam_access(crond:account): auth could not identify password for [root]
Jun 24 10:45:01 server1 crond[22405]: pam_access(crond:account): auth could not identify password for [admin]
Jun 24 10:45:01 server1 crond[22400]: pam_localuser(crond:account): auth could not identify password for [root]

Similarly /var/log/cron.log is showing that PAM is failing:

Jun 24 12:40:01 server1 crond[26129]: (admin) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26129]: (admin) FAILED to authorize user with PAM (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26130]: (admin) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26130]: (admin) FAILED to authorize user with PAM (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26125]: (root) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26125]: (root) FAILED to authorize user with PAM (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26127]: (root) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26127]: (root) FAILED to authorize user with PAM (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26131]: (admin) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26131]: (admin) FAILED to authorize user with PAM (Authentication information cannot be recovered)

I've tried the following with no success:

  • Rebooting the server
  • Restarting the cron service
  • Editing /etc/security/access.conf to ensure that root is allowed access to the cron
  • cron.allow is non-existent and cron.deny is empty so that shouldn't be the problem
  • Disabling SELinux and rebooting
  • Changing root password to ensure it's not an expiry issue
  • Checked /etc/passwd and /etc/shadow for passwords of both root and admin user
  • Removed all cron jobs except a simple one to write to a text file every minute. This cron job also did not work so it's not related to the jobs in the cron.

Please help as I'm not sure what else to do to fix this problem. It's worth noting that this issue started on June 21 and no changes were made to the server when it began occurring.

|improve this question
  • 1
    Where either users' passwords changed recently? Can you verify the passwords for root, and admin in both /etc/passwd, and /etc/shadow? In /etc/passwd, there should be an x for the password field, and in /etc/shadow there should be a big long hash. – Lucas Ramage Jun 24 at 13:32
  • @LucasRamage Yes the passwords are there for both users in both files you mentioned. I changed the root password to ensure it wasn't a password expiry issue – pauloz1890 Jun 24 at 13:41
  • Were any relevant system packages updated just before or on June 21st? And, could you check if any file inside /etc/pam.d/ was recently changed? – Edward Jun 24 at 13:46
  • @Edward I checked the files inside that folder and nothing was changed recently. It's possible that WHM did some upgrades but nothing was changed or upgraded manually. I know the exact timestamp the problem began as I can see it in the logs. I'm looking for which files changed around this time. – pauloz1890 Jun 24 at 13:56
  • Here is the log from when it all began: imgur.com/a/eRwAPMm – pauloz1890 Jun 24 at 14:04
1

I was able to solve the issue. It is related to a file called /lib/libgrubd.so.

If you're experiencing this issue then check /etc/ld.so.preload. If this file contains /lib/libgrubd.so (it may be the only line in that file) then remove that line and PAM should start working again. I also removed the /lib/libgrubd.so file from the system as it may be associated with a virus as shown here.

Still not entirely sure what caused the issue but this was the reason causing PAM to functioning incorrectly. See more info here.

|improve this answer
  • Your cpanel server could be hacked due to CVE-2019-10149 Exim, update cpanel to prevent this issue, check documentation.cpanel.net/display/CKB/CVE-2019-10149+Exim – garconcn Jun 26 at 17:15
  • Thanks. I was on v78 when it occurred. I already updated to v80 a few days ago. How can I tell if my system was affected in any other way? – pauloz1890 Jun 26 at 23:33
  • sorry, I don't have more info. If you own the server, it's best to contact cpanel support directly at tickets.cpanel.net, they are very good. I am also surprised you had these files even though you are using latest version. – garconcn Jun 27 at 0:13
1

I just faced the same issue. What they tell me is this:

I regret to inform you that this server is compromised at the root level by malware known as ShellBot. This malware is known to cause errors when running the "crontab" command but can potentially cause many other problems including not being able to start certain services.

The presence of /lib/libgrubd.so (which is not normally found on clean systems) is an indication of this malware. This malware is installed in a way that causes it to execute every time a program is run on the server, and can potentially cause aberrant behavior of any process.

|improve this answer
  • Thanks for sharing that. I scanned with ClamAV and my system is clean. – pauloz1890 Jun 27 at 14:48
  • Who is "they"? – Wildcard Jul 19 at 2:03

protected by Community♦ Jul 18 at 20:21

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.