How about sharing your favorite lessons learned moments?
locked by Michael Mrozek♦ Sep 16 '11 at 12:17
This question exists because it has historical significance, but it is not considered a good, on-topic question for this site so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. See the help center for guidance on writing a good question.
Read more about locked posts here.
On a Debian installation back in 1999. 14 floppy disks for the basic installation.
I tried to get xfree86 working.
X didn't start.
I had to find out the settings of my graphics card (memory, horizontal and vertical refresh rate), which was completely undocumented.
It turned out
superprobe found out the correct amount of internal graphics card memory (1024 kB).
But it took me nearly a week to find out that the resolution setting (1024x768) didn't work. I had to switch it to 640x480 until the graphics card finally worked (at 1024x768... buuuuuug....).
I tried to get the serial port mouse to work on COM1. So I tried to get the mouse to work. Reading a book (back then I had no usable high-speed internet), I tried with
And it didn't work and didn't work.
It took me nearly another week to find out this was because I needed to type the
S is uppercase, not lowercase...
It was about then when I finally realized what 'case-sensitive' really means.
A long time ago, I was installing MkLinux on my Mac, and I wanted to replace the file that governed command processing (not the shell, something more basic, don't remember quite what anymore). The instructions said to do
mv x y, so I decided to be cautious and started with
rm y. The intention was to
mv x y afterwards, but of course that didn't work. I reinstalled.
chown nobody:nobody /*
As root of course....
chmod -R 777 /: after that I couldn't figure out how to restore the permission on the whole filesystem and then I reinstalled the OS.
Never did that again (and still don't know how to restore such a situation).
I mapped the CapsLock to ESC on the entire system. When I did it, the Capslock was on.
A reboot removed the permanent state of CapsLock. It was mapped to ESC
It wasn't really painful but I felt stupid when I realized what I'd done!
I don't keep track of errors on my own boxes, but from the last 15 years here are my two work fatalities:
1995: Standard '
rm -rf as root' failure on a single box (not my design!) acting as a combined NIS master server + sole DNS server for the company + primary SMTP/POP3 server for the company. It was SunOS 4.1.3_U1 as I recall. Unsurprisingly, like the witness to a horiffic crime, I do not remember the ensuing 48 hours.
1998: Ran newfs (SunOS) on a production AFS (now OpenAFS) file server instead of the replacement we were standing up. Spent the afternoon and night restoring from tape.
One time, I had defined an alias to help me clean temp files:
alias clean_dir_tmp="find /tmp -maxdepth 1 -user **** | grep *** | xargs rm -rf"
and of course one day, when I had forget what the alias was doing I mistyped:
$ clean_dir_tmp *
I think I lost a good couple of seconds before I realized what was happening... :(
I was working as usual with my Mac laptop and my Ubuntu desktop. When I plugged the external HD of the Mac (HFS+ filesystem) in Ubuntu, I noticed that the owner was
My UID was different on my Mac and on my desktop so since I wanted a nicer output with
ls -l, I decided to change that.
So I modified the UID on the mac and launched a big
chown -R * on the HD from Ubuntu. The only thing that I didn't know was that the HFS+ driver for Linux was not stable. To this day, I haven't been able to mount this HD on Linux or my Mac again...
dist-upgrade to unstable (or was it testing?) on a remote production server.
Ignored warning about
libc switching to
Not sure where installation actually failed, but I was left with one root console on a dial-up line with every single app exploding. Just one running
Had a lot of fun of recovering it. Uploaded statically linked
dpkg, rolled back
libc, built a custom kernel with RAID support. I think that took around 3 hours with my dial-up line. When I finally rebooted it,
fsck took like 10 minutes. Quite painful 10 minutes I should say.
Somehow managed to unmount /dev and thought I would be screwed forever if I rebooted the machine.
Nerve wracking hour ensued trying to figure out if it would be safe to reboot it. It was, nothing bad happened.
Wanted to make an archive:
tar -cvzf mydir/* mydir.tar.gz
mydir/* expanded to
Remember that the archive name follows the
-f option of tar!
My favorite is when I was building the new Solaris system for an Oracle database installation. Everything was in place including the high priced oracle consultant right in the middle of the DB optimization work he was doing. I was in the server room checking on another server, when I tripped over the power cord to the server.
I was once trying to zero out a USB thumbdrive using
Needless to say, when tty1 started spitting out ReiserFS errors from my root partition, I had to reinstall...
Trying to get the Xwindows driver for my Nvidia card working when Fedora initially released the Nouveau driver. I had downloaded the Nvidia source to compile and install myself as I had many times in the past, but this release, I could just not get it to work. There were quite a few steps to find in the Fedora Forums to completely disable the Nouveau driver, and get the Nvidia driver working. Quite painful to say the least.
Back in the day you had to do erase the first 512 bytes of a partition to properly format FAT drives from Linux. This is done using the
dd if=/dev/zero of=/dev/hda1 bs=512 count=1
Except the FAT partition was
I didn't realize what had happened until after I rebooted. Luckily I was able to recover it by re-installing Lilo, or something.
Twenty minutes ago, I was painstakingly recreating a complex directory structure from files I had on my other PC. I decided to run
du to see if it was near completion. Given the size, I knew it would take few minutes, so went to get some coffee.
On my return I noticed, to my severe dismay, that instead of running
du -hs /path/to/important/folder
I had absent-mindedly run
rm -r /path/to/important/folder
Many web apps like magento or eZ Publish have a
var/cache/ folder and a way to clear the cache that is faster than going in the app backend is to do this:
rm -rf var/cache
After doing this a couple of times, it is scientifically proven that you always end up:
- Either doing this in the root of your server; or
- Adding a
Lesson learned: create an alias for this command with an absolute path in it.
The first time I installed GNU Linux on my desktop, I installed Debian, no help, I only installed the basic system, no GUI. And I was like:
"OMFG OMFG WHAT DO I HAVE TO DO NOW?... I will need to go back to Windows"
But then I remembered how to install packages (first time in GNU Linux, only a
jose@debian:$ output and only read some things from Debian) and the name of an IRC Client: IRSSI and the name: GNOME and then I installed them... since that day, I install IRSSI and GNOME in every machine...
It feels... good experimenting with your computer xD
1st ex employee: How do I do (something trivial)?
2nd ex employee:
sudo rm -rf /
1st ex employee: Haha ok
1st ex employee (Having forgotten that he'd just
sudoed something else, so it didn't ask for his password again) : oh F**K!!!
Installing Debian (with net-install) on a computer and realise after rebooting that I had somehow skipped the part where one is supposed to choose which packages to install.
Sure, a non-graphical system with basically only
cd is working just fine ;).
I wanted to delete a file and its backup copy (
file~) on a SuSE system I rarely used and didn't notice that command completion was configured differently. I typed
rm fi<tab> and expected the cursor to be after the last letter. The system ignored
~ files and inserted a space after the file name.
So I wanted to type
and I did
rm file *
Now I am always checking what the suggested completion is.
A while ago, I needed to do some extensive configuration on one of my machines, that mostly involved editing bunch of files in
/etc. I decided to be really careful about it, so I created an
etc directory in my
$HOME, copied the files that needed editing in there, spent couple hours doing the edits. I carefully checked all the files, made sure that all the edits were exactly the way they were supposed to be, logged in as root and copied the edited files back to
/etc. By then, it was very late at night. Still as root, I decided to clean up, and instead of
rm -rf etc, I typed in
rm -rf /etc. I did not get much sleep that night.
I was on the phone with a colleague who was out at a customer site. She was working on their systems and I was telling her stroke for stroke what to type, she misheard me when I told her
rm -rf .??* and typed
rm -rf .?*. But she wasn't in the directory she had told me, she was in the root directory. Wiped not only the dot files, but the entire OS.
Back when I was "learning as a sysadmin", I was writing my own adduser script (didn't have it on early SysV). A shell error in the script
(cat /etc/passwd; echo ...) > /etc/passwd which of course wiped the passwd file and then, by accident, I hit Ctrl-D to exit the su shell. Had to go in to the office at 1am Saturday morning to get the boot diskettes.
chown -R /usr. Somehow, I thought it was a good idea.
Subsequently, a puppet script did some bad stuff, and somehow I lost
Our qualified sysadmin wasn't able to recover my
sudo rights, and I had to re-image the machine.