What are the recommended ways of defending a remote *nix install from a hamfisted admin?

Question

Once in a while (often a long while), I have a difficulty where I execute a command that completely screws up a Linux machine.

Most recently, I accidentally mounted-again the root partition (thinking it was the new USB drive I had just formatted), and then proceeded to recursively chown the partition to myself (again, just trying to grant myself user-access to the USB drive). As soon as I realized what I had done (in mid-progress), I aborted it, but the damage was done. Many core programs were no longer root owned, so the machine was essentially in a zombified state. Some user functions (ssh, rsync) still functioned, but administration level stuff was totally locked-out. Couldn't mount, umount, reattach to screen sessions, reboot, etc.

If the machine were in the living room here with me, "repairing" it (reinstall) would have been trivially easy. But it isn't. It is in my brother's house. He's not big on me walking him through repairs/reinstall, and I understand that. So, I'm going over in a few days to fix the damage I did (and hopefully install something more admin-screwup-resistant).

I say all that to ask the question: What are the recommended ways of hardening an install against admin-hamfistedness?

Things not considered, or considered and dropped quickly:

1. Harden the administrator to not execute stupid commands: A great idea, but won't work, because as a human, I occasionally will do things that I realize after-the-fact are a bad idea. What I'm looking to do is out-think myself in advance, so when I do something stupid, the machine will refuse, and I'll realize "Oh crap! That could have been Very Bad (TM)! Let's not do that again."

Things I've considered:

1. Mount the root partition read-only: Would protect the root from changes, which might have negative affects if parts are expected to be writeable, and aren't. Also wouldn't necessarily protect the partition from being mounted again somewhere else as read-write.
2. Use a compressed-readonly root image of some sort with a union-like writeable layer above it so no changes are ever really made to root, and a reboot clears any screw-ups: This would be OK/good if no changes ever need to be made to root, and maybe /etc could be reloaded/populated from a persistent file somewhere else.
3. Use btrfs with regular (daily, maybe) snapshots, so that if an error is made, recovery is easier: Might still be sub-optimal as it would require direct user intervention, and I don't know that I could walk someone else through the changes to roll back the oops.
4. Use a more "live"/"embedded" Linux/BSD distro designed more with stability/predictability/security in mind instead of a more generic distro

As things stand now, I'm likely to use option 4 to install a somewhat more limited system than the full Debian install I had been using. But as just a file server and torrent client, it should work fine, and as a remote machine, defending the machine from myself is a pretty big asset.

Answer 1

The harsh truth is that nothing can protect you from your own stupidity. There's no DWIM (do what I mean) interface. The computer can't tell the difference between what is intentional and what is accidental. No matter how much abstraction you pile on the wrong stray command can destroy it all.

The simple answer is to slow down and pay attention to what you're doing.

Answer 2

Run your installation in a virtual machine. Take a snapshot of a known good state. Take snapshots before doing anything risky. Do almost nothing in the host environment. If you screw up, connect to the host environment and restore the snapshot.

Answer 3

Nothing prevents you from shooting yourself in the foot. You "thought" the root partition is a USB stick. You could just as easily mistake an important machine for a disposable VM.(Happens to the best of us)

What is important is to make the service that your computers provide, redundant.

In this case, you could have two Linux versions installed on two separate partitions. You can tell your brother to boot into the other one.(Just an idea)

What is most important that you take backups, and have a restore strategy.

In this case, since you have taken the responsibility of your brothers PC, you should take continuos backups of whatever data you can, and keep multiple copies with you.

You can also provide your brother a USB Linux drive to boot from, with a SSH server and password set. And set his PC to boot from USB. Then in an emergency, just ask him to insert the USB stick and restart the PC.

Answer 4

One small, but possibly very helpful step in that direction is to install molly-guard which will prevent you from accidentally calling reboot or even remote on the remote host. It detects whether you are logged in from remote and requires you to type the hostname to confirm your action.

Answer 5

Don't do things as root user. Set up sudo to allow your regular account to do root things, but with a password. This gives you one last chance to see what you are really doing.

But when you do run as root, set up aliases for common commands that force interactive use. e.g. alias rm="rm -i" will make rm prompt before removal. You can the explicitly override with -f (a conscious decision) if you really want to rm * (would then be rm -f *).

You didn't say what FS was on the USB. Usually they are VFAT. You can mount these with options to make every file already appear to be owned by a specific user. Then you never actually have to run chown -r ... and thus eliminate the possibility of mistake.

Make your root shell prompt red colored, to remind you that you are running with elevated privilege.

Generally, makes things hard for you to do as root, with obstacles such as password prompts, etc.

Now, after the fact to fix it you can get access to another machine like it, and use find to show you the SUID/SGID programs. Then make the damaged disk match that one with the chmod command.