Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Once in a while (often a long while), I have a difficulty where I execute a command that completely screws up a Linux machine.

Most recently, I accidentally mounted-again the root partition (thinking it was the new USB drive I had just formatted), and then proceeded to recursively chown the partition to myself (again, just trying to grant myself user-access to the USB drive). As soon as I realized what I had done (in mid-progress), I aborted it, but the damage was done. Many core programs were no longer root owned, so the machine was essentially in a zombified state. Some user functions (ssh, rsync) still functioned, but administration level stuff was totally locked-out. Couldn't mount, umount, reattach to screen sessions, reboot, etc.

If the machine were in the living room here with me, "repairing" it (reinstall) would have been trivially easy. But it isn't. It is in my brother's house. He's not big on me walking him through repairs/reinstall, and I understand that. So, I'm going over in a few days to fix the damage I did (and hopefully install something more admin-screwup-resistant).

I say all that to ask the question: What are the recommended ways of hardening an install against admin-hamfistedness?

Things not considered, or considered and dropped quickly:

  1. Harden the administrator to not execute stupid commands: A great idea, but won't work, because as a human, I occasionally will do things that I realize after-the-fact are a bad idea. What I'm looking to do is out-think myself in advance, so when I do something stupid, the machine will refuse, and I'll realize "Oh crap! That could have been Very Bad (TM)! Let's not do that again."

Things I've considered:

  1. Mount the root partition read-only: Would protect the root from changes, which might have negative affects if parts are expected to be writeable, and aren't. Also wouldn't necessarily protect the partition from being mounted again somewhere else as read-write.
  2. Use a compressed-readonly root image of some sort with a union-like writeable layer above it so no changes are ever really made to root, and a reboot clears any screw-ups: This would be OK/good if no changes ever need to be made to root, and maybe /etc could be reloaded/populated from a persistent file somewhere else.
  3. Use btrfs with regular (daily, maybe) snapshots, so that if an error is made, recovery is easier: Might still be sub-optimal as it would require direct user intervention, and I don't know that I could walk someone else through the changes to roll back the oops.
  4. Use a more "live"/"embedded" Linux/BSD distro designed more with stability/predictability/security in mind instead of a more generic distro

As things stand now, I'm likely to use option 4 to install a somewhat more limited system than the full Debian install I had been using. But as just a file server and torrent client, it should work fine, and as a remote machine, defending the machine from myself is a pretty big asset.

share|improve this question
2  
Backups! Backups fix everything you break! (Just make sure you don't break the backups) –  Malfist Jul 3 '12 at 14:56

5 Answers 5

up vote 10 down vote accepted

Run your installation in a virtual machine. Take a snapshot of a known good state. Take snapshots before doing anything risky. Do almost nothing in the host environment. If you screw up, connect to the host environment and restore the snapshot.

share|improve this answer
    
Of course that doesn't prevent me from screwing up the host environment by issuing the wrong command there... Otherwise, not a bad idea. But also requires virtualization-capable hardware, and ends up reducing the quantity of ram available to the machine I'm looking to use... Some good upsides, but a few too many downsides, I think. –  killermist Jul 2 '12 at 23:15
7  
Do almost nothing in the host environment. Nowadays, there aren't many non-virtualization-capable processors still running, and RAM isn't very expensive. –  Gilles Jul 2 '12 at 23:21
    
It's very common for me to install Xen on a new server only to run one paravirtualized guest that is afforded most of the available resources (beyond what dom-0 / the host OS requires). This saves me from having to call on (expensive) remote hands when things go wrong, no matter who or what caused it :) Xen is well supported by Debian. –  Tim Post Jul 3 '12 at 1:51
    
@TimPost I remember looking at Xen before. But it looked VERY complicated to get going. Could you recommend a good "getting started" guide for Xen? I've used Proxmox (KVM/OpenVZ) before, but sometimes it ends up being a nuisance trying to set some things up. –  killermist Jul 3 '12 at 2:25

One small, but possibly very helpful step in that direction is to install molly-guard which will prevent you from accidentally calling reboot or even remote on the remote host. It detects whether you are logged in from remote and requires you to type the hostname to confirm your action.

share|improve this answer
    
This isn't a horrible idea. But I knew full-well where I was, and what I was trying to do. I additionally admit that Mollyguard on some machines has prevented me from doing something stupid that I should have known better. This won't help directly with the hamfisted-ness of this admin, but +1 for good ideas. –  killermist Jul 3 '12 at 17:10

Don't do things as root user. Set up sudo to allow your regular account to do root things, but with a password. This gives you one last chance to see what you are really doing.

But when you do run as root, set up aliases for common commands that force interactive use. e.g. alias rm="rm -i" will make rm prompt before removal. You can the explicitly override with -f (a conscious decision) if you really want to rm * (would then be rm -f *).

You didn't say what FS was on the USB. Usually they are VFAT. You can mount these with options to make every file already appear to be owned by a specific user. Then you never actually have to run chown -r ... and thus eliminate the possibility of mistake.

Make your root shell prompt red colored, to remind you that you are running with elevated privilege.

Generally, makes things hard for you to do as root, with obstacles such as password prompts, etc.

Now, after the fact to fix it you can get access to another machine like it, and use find to show you the SUID/SGID programs. Then make the damaged disk match that one with the chmod command.

share|improve this answer
    
These aren't horrible recommendations, but already in use as default. I was acting using sudo chmod. And since I had just used sudo to partition and format the drive (as btrfs), it remembered my admin-ness, and didn't ask for the password again. Also, part of it was that I had misread the results of ls -lh /dev/disks/by-uuid/ and put the wrong UUID into /etc/fstab for /volumes/temp when mounting. –  killermist Jul 3 '12 at 15:58
    
@killermist Well then I guess you were, subconsciously, bound and determined to mess up your brother's computer. ;-) –  Keith Jul 4 '12 at 11:14

Nothing prevents you from shooting yourself in the foot. You "thought" the root partition is a USB stick. You could just as easily mistake an important machine for a disposable VM.(Happens to the best of us)

What is important is to make the service that your computers provide, redundant.

In this case, you could have two Linux versions installed on two separate partitions. You can tell your brother to boot into the other one.(Just an idea)

What is most important that you take backups, and have a restore strategy.

In this case, since you have taken the responsibility of your brothers PC, you should take continuos backups of whatever data you can, and keep multiple copies with you.

You can also provide your brother a USB Linux drive to boot from, with a SSH server and password set. And set his PC to boot from USB. Then in an emergency, just ask him to insert the USB stick and restart the PC.

share|improve this answer
    
It actually is MY machine but is running on his network (more bandwidth, and whatnot). The idea of leaving behind a bootable USB drive from which I can repair the machine without him having to do anything but plug/reboot is a great idea. I'd also have to double check the bios configuration to make sure it will boot USB if available before HDD, which with some bios's, can be a pain. Definitely worth additional study/investigation. –  killermist Jul 3 '12 at 16:04

The harsh truth is that nothing can protect you from your own stupidity. There's no DWIM (do what I mean) interface. The computer can't tell the difference between what is intentional and what is accidental. No matter how much abstraction you pile on the wrong stray command can destroy it all.

The simple answer is to slow down and pay attention to what you're doing.

share|improve this answer
1  
A useful point, to be sure. But I know that sooner or later, I'm going to commit an oops. If I didn't, I'd have to start doubting my humanity. I appreciate the concept, and I've coached myself about it after a few failures, but at this point, I'm looking more to prevent/mitigate damage, because I know that sooner or later hamfisted-admin errors WILL occur. I can't really change the nature of this beast (me). +1 for good wisdom. –  killermist Jul 3 '12 at 2:33
2  
Make no assumptions, have a healthy dose of paranoia, issue commands like your life depended on them, pause full-beat before you press Enter and think about the command and its side-effects. And have backups for when this inevitably fails to protect you. –  Alexios Jul 3 '12 at 5:47
1  
Backups, backups, backups. And use sudo for that slight additional edge of asking for your password again before executing a command. –  Shadur Jul 3 '12 at 8:28
    
@Alexios I CAN NOT be that serious about administration of "my own" machines. If it turned into that, I would have to sell off all my hardware, and buy a few tablets and turn into a typical American vegetable that doesn't care about how things work. Work/client machines, that mindset is useful, but if I have to treat my own machines like that, there is no joy in it, and with no joy, there is NO REASON to have it. –  killermist Jul 3 '12 at 16:29
    
@Alexios One of the things my Dad did right was to instill in me a fearless-ness of "No matter what you do, you can't hurt this machine." (granted, that was on a TRS-80 Color Computer 2, which was pretty bullet-proof), but that mindset has stuck. If I'm having to second-guess every action, then it isn't fun/learning. At that point, it is a chore, with no pay-off. –  killermist Jul 3 '12 at 16:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.