32

I'm getting the following error from sudo:

$ sudo ls
sudo: /etc/sudoers is owned by uid 1000, should be 0
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin

Of course I can't chown it back to root without using sudo. We don't have a password on the root account either.

I honestly don't know how the system got into this mess, but now it's up to me to resolve it.

Normally I would boot into recovery mode, but the system is remote and only accessible over a VPN while booted normally. For the same reason, booting from a live CD or USB stick is also impractical.

The system is Ubuntu 16.04 (beyond EOL, don't ask), but the question and answers are probably more general.

5
  • 4
    I doubt there is a general solution to this problem. If there was, then why couldn't I employ it on systems where I have a user account (without sudo privileges by design) to grant myself sudo privileges? There might be a solution for your particular problem but it would depend on unstated details and it would not generalize.
    – emory
    Nov 16, 2021 at 14:31
  • This is why I always leave my editor open after saving a sudoers file. I check in another CLI that I can still sudo ls to avoid getting locked-out. If the sudoers file is broken, I'm still editing it in another process and can fix the error. Nov 17, 2021 at 20:11
  • 1
    @ChristopherSchultz Use visudo instead, it does all that for you.
    – Thomas
    Nov 18, 2021 at 7:53
  • @Thomas That should be the correct answer. Ironically, the same day I posted that comment, I locked myself out of an EC2 instance with a typo in a sudoers file that had been copied-into /etc/sudoers.d/ by a script I was working on. I had to move the root volume to another EC2 instance to rescue myself. 🤦 Nov 18, 2021 at 21:55
  • visudo does not protect against broken automation. Nor against disk corruption -- as turned out to be the case in my situation, but even that was hard to figure out without root access :)
    – Thomas
    Feb 23, 2022 at 9:52

3 Answers 3

58

The procedure described here (which may itself be an imperfect copy of this Ask Ubuntu answer) performed the miracle. I'm copying it here, and adding some more explanations.

Procedure

  1. Open two SSH sessions to the target server.

  2. In the first session, get the PID of bash by running:

     echo $$
    
  3. In the second session, start the authentication agent with:

     pkttyagent --process 29824
    

    Use the pid obtained from step 1.

  4. Back in the first session, run:

     pkexec chown root:root /etc/sudoers /etc/sudoers.d -R
    
  5. Enter the password in the second session password promt.

Explanation

Similar to sudo, pkexec allows an authorized user to execute a program as another user, typically root. It uses polkit for authentication; in particular, the org.freedesktop.policykit.exec action is used.

This action is defined in /usr/share/polkit-1/actions/org.freedesktop.policykit.policy:

  <action id="org.freedesktop.policykit.exec">
    <description>Run programs as another user</description>
    <message>Authentication is required to run a program as another user</message>
    <defaults>
      <allow_any>auth_admin</allow_any>
      <allow_inactive>auth_admin</allow_inactive>
      <allow_active>auth_admin</allow_active>
    </defaults>
  </action>

auth_admin means that an administrative user is allowed to perform this action. Who qualifies as an administrative user?

On this particular system (Ubuntu 16.04), that is configured in /etc/polkit-1/localauthority.conf.d/51-ubuntu-admin.conf:

[Configuration]
AdminIdentities=unix-group:sudo;unix-group:admin

So any user in the group sudo or admin can use pkexec.

On a newer system (Arch Linux), it's in /usr/share/polkit-1/rules.d/50-default.rules:

polkit.addAdminRule(function(action, subject) {
    return ["unix-group:wheel"];
});

So here, everyone in the wheel group is an administrative user.

In the pkexec manual page, it states that if no authentication agent is found for the current session, pkexec uses its own textual authentication agent, which appears to be pkttyagent. Indeed, if you run pkexec without first starting the pkttyagent process, you are prompted for a password in the same shell but it fails after entering the password:

polkit-agent-helper-1: error response to PolicyKit daemon: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie

This appears to be an old bug in polkit that doesn't seem to be getting any traction. More discussion.

The trick of using two shells is merely a workaround for this issue.

8
  • I've tried this on an AWS centos7 instance, and the user prompt in the second SSH session is for Authenticating as: Cloud User (centos) Is there a way to set the user used ? /etc/polkit-1/localauthority.conf.d/ is empty for me.
    – Criggie
    Nov 15, 2021 at 20:37
  • @Criggie You should be able to do a ssh <user>@<host> to define the user you are ssh'ing as.
    – B.Kaatz
    Nov 15, 2021 at 20:39
  • 1
    Well, by default, pkexec is supposed to run the requested command as root if no username is supplied. So, how about trying pkexec --user root chown root:root /etc/sudoers /etc/sudoers.d -R?
    – B.Kaatz
    Nov 15, 2021 at 20:52
  • 1
    @Criggie I got a prompt to choose one of the 3 users that had the required permissions. Also I think the configuration format has changed from INI syntax to JavaScript. Check /usr/share/polkit-1/rules.d/50-default.rules or search for addAdminRule in that directory.
    – Thomas
    Nov 16, 2021 at 9:47
  • 1
    @OlivierDulac FYI, PolicyKit/polkit has nothing to do with systemd and predates it by almost half a decade. It's been around for about 15 years now. If you have ever used some GUI system administration application (package manager etc.), polkit was the mechanism used to do administrative tasks without you having to run the GUI as root by hand.
    – TooTea
    Nov 16, 2021 at 20:46
6

Boot into a live environment and fix it from there.

This obviously requires either physical access of ‘physical-equivalent’ access (if dealing with a VM or VPS), but it almost always works, doesn’t care about what init system you have, and does not care about whether or not you have a root password.

The general approach is relatively simple:

  1. Prepare a live CD (or other boot media) that is compatible with the OS you are using (mostly, this just comes down to supporting the particular combination of storage drivers and filesystems needed to mount the root filesystem).
  2. Boot the affected system using that boot media.
  3. Mount the root filesystem of the affected system somewhere.
  4. Edit and save the file.
  5. Reboot.

This kind of fix works because it completely sidesteps the access controls of the affected system, letting you do pretty much whatever you want. This level of unrestricted access to the system is part of why physical security is so important.

Note that if taking this approach, you may need to do something special when rebooting back into the ‘normal’ system to make things work properly. On a stock Ubuntu install this should not be needed, but if you’re using SELinux (or even less likely, IMA and EVM), you may need to add extra boot options and then run a command from the booted system to fix security labels.


For those who do have a root password, just use single-user mode.

Systemd calls this ‘rescue mode’, but everyone else just calls it single-user mode. This essentially boots to a root shell with almost nothing but critical services running. It’s traditionally used for fixing exactly this type of issue.

The one downside to this is that on any properly secured modern system, single-user mode is password protected and requires being able to log in as the root user (the reason for this being that access to the system console does not inherently imply physical access to the system hardware, so some authentication should be used).

1
  • 4
    You don't need single-user mode if you have the root password; just use su.
    – chepner
    Nov 17, 2021 at 23:24
-1

Whenever you plan to experiment with sudo config, it's a good idea to keep another root terminal open: in case you break something, you will already have the rights to fix it.

If it's not possible to keep a terminal open (e.g. your experiments involve a reboot), set an actual password for the root user. Then if sudo is broken, you can simply login as root (open new console or run su) and fix you system.

4
  • This is good advice, but does not answer the question.
    – Thomas
    Nov 18, 2021 at 14:47
  • @Thomas Sure, but considering how popular this question is, I believe people should know they don't have to hack their way out of a broken sudo configuration. Nov 18, 2021 at 14:52
  • That's what the comment system is for :)
    – Thomas
    Nov 18, 2021 at 15:20
  • @Thomas Not really, comments cannot be downvoted or fixed if they are wrong, and can be deleted or moved to chat even if they are correct. Nov 18, 2021 at 15:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .