What are your favorite *painful* Unix moments

Question

How about sharing your favorite lessons learned moments?

Answer 1

On a Debian installation back in 1999. 14 floppy disks for the basic installation.

I tried to get xfree86 working. But X didn't start. I had to find out the settings of my graphics card (memory, horizontal and vertical refresh rate), which was completely undocumented. It turned out superprobe found out the correct amount of internal graphics card memory (1024 kB).

But it took me nearly a week to find out that the resolution setting (1024x768) didn't work. I had to switch it to 640x480 until the graphics card finally worked (at 1024x768... buuuuuug....).

I tried to get the serial port mouse to work on COM1. So I tried to get the mouse to work. Reading a book (back then I had no usable high-speed internet), I tried with

/dev/ttys0

And it didn't work and didn't work. It took me nearly another week to find out this was because I needed to type the S is uppercase, not lowercase...

/dev/ttyS0

It was about then when I finally realized what 'case-sensitive' really means.

Answer 2

A long time ago, I was installing MkLinux on my Mac, and I wanted to replace the file that governed command processing (not the shell, something more basic, don't remember quite what anymore). The instructions said to do mv x y, so I decided to be cautious and started with rm y. The intention was to mv x y afterwards, but of course that didn't work. I reinstalled.

Answer 3

chown nobody:nobody /*

As root of course....

Answer 4

Mine was chmod -R 777 /: after that I couldn't figure out how to restore the permission on the whole filesystem and then I reinstalled the OS.

Never did that again (and still don't know how to restore such a situation).

Answer 5

I mapped the CapsLock to ESC on the entire system. When I did it, the Capslock was on.

A reboot removed the permanent state of CapsLock. It was mapped to ESC

It wasn't really painful but I felt stupid when I realized what I'd done!

Answer 6

I don't keep track of errors on my own boxes, but from the last 15 years here are my two work fatalities:

1995: Standard 'rm -rf as root' failure on a single box (not my design!) acting as a combined NIS master server + sole DNS server for the company + primary SMTP/POP3 server for the company. It was SunOS 4.1.3_U1 as I recall. Unsurprisingly, like the witness to a horiffic crime, I do not remember the ensuing 48 hours.

1998: Ran newfs (SunOS) on a production AFS (now OpenAFS) file server instead of the replacement we were standing up. Spent the afternoon and night restoring from tape.

Answer 7

First one

One time, I had defined an alias to help me clean temp files:

alias clean_dir_tmp="find /tmp -maxdepth 1 -user **** | grep *** | xargs rm -rf"

and of course one day, when I had forget what the alias was doing I mistyped:

$ clean_dir_tmp *

I think I lost a good couple of seconds before I realized what was happening... :(

Second one

I was working as usual with my Mac laptop and my Ubuntu desktop. When I plugged the external HD of the Mac (HFS+ filesystem) in Ubuntu, I noticed that the owner was ?????. My UID was different on my Mac and on my desktop so since I wanted a nicer output with ls -l, I decided to change that.

So I modified the UID on the mac and launched a big chown -R * on the HD from Ubuntu. The only thing that I didn't know was that the HFS+ driver for Linux was not stable. To this day, I haven't been able to mount this HD on Linux or my Mac again...

Answer 8

Debian dist-upgrade to unstable (or was it testing?) on a remote production server.

Ignored warning about libc switching to nptl threads.

Not sure where installation actually failed, but I was left with one root console on a dial-up line with every single app exploding. Just one running ssh and bash.

Had a lot of fun of recovering it. Uploaded statically linked dpkg, rolled back libc, built a custom kernel with RAID support. I think that took around 3 hours with my dial-up line. When I finally rebooted it, fsck took like 10 minutes. Quite painful 10 minutes I should say.

Answer 9

Somehow managed to unmount /dev and thought I would be screwed forever if I rebooted the machine.

Nerve wracking hour ensued trying to figure out if it would be safe to reboot it. It was, nothing bad happened.

Answer 10

Wanted to make an archive:

tar -cvzf mydir/* mydir.tar.gz

Of course, mydir/* expanded to mydir/myfile.cpp mydir/myfile.h

Remember that the archive name follows the -f option of tar!

Answer 11

My favorite is when I was building the new Solaris system for an Oracle database installation. Everything was in place including the high priced oracle consultant right in the middle of the DB optimization work he was doing. I was in the server room checking on another server, when I tripped over the power cord to the server.

Answer 12

I was once trying to zero out a USB thumbdrive using dd.

Needless to say, when tty1 started spitting out ReiserFS errors from my root partition, I had to reinstall...

Answer 13

Trying to get the Xwindows driver for my Nvidia card working when Fedora initially released the Nouveau driver. I had downloaded the Nvidia source to compile and install myself as I had many times in the past, but this release, I could just not get it to work. There were quite a few steps to find in the Fedora Forums to completely disable the Nouveau driver, and get the Nvidia driver working. Quite painful to say the least.

Answer 14

Back in the day you had to do erase the first 512 bytes of a partition to properly format FAT drives from Linux. This is done using the dd command.

dd if=/dev/zero of=/dev/hda1 bs=512 count=1

Except the FAT partition was /dev/hdb1

I didn't realize what had happened until after I rebooted. Luckily I was able to recover it by re-installing Lilo, or something.

Answer 15

Twenty minutes ago, I was painstakingly recreating a complex directory structure from files I had on my other PC. I decided to run du to see if it was near completion. Given the size, I knew it would take few minutes, so went to get some coffee.

On my return I noticed, to my severe dismay, that instead of running

du -hs /path/to/important/folder

I had absent-mindedly run

rm -r  /path/to/important/folder

Answer 16

Many web apps like magento or eZ Publish have a var/cache/ folder and a way to clear the cache that is faster than going in the app backend is to do this:

rm -rf var/cache

After doing this a couple of times, it is scientifically proven that you always end up:

• Either doing this in the root of your server; or
• Adding a / before var

Lesson learned: create an alias for this command with an absolute path in it.

Answer 17

The first time I installed GNU Linux on my desktop, I installed Debian, no help, I only installed the basic system, no GUI. And I was like:

"OMFG OMFG WHAT DO I HAVE TO DO NOW?... I will need to go back to Windows"

But then I remembered how to install packages (first time in GNU Linux, only a jose@debian:$ output and only read some things from Debian) and the name of an IRC Client: IRSSI and the name: GNOME and then I installed them... since that day, I install IRSSI and GNOME in every machine...

It feels... good experimenting with your computer xD

Answer 18

1st ex employee: How do I do (something trivial)?
2nd ex employee: sudo rm -rf /
1st ex employee: Haha ok
...
1st ex employee (Having forgotten that he'd just sudoed something else, so it didn't ask for his password again) : oh F**K!!!

Answer 19

Installing Debian (with net-install) on a computer and realise after rebooting that I had somehow skipped the part where one is supposed to choose which packages to install.

Sure, a non-graphical system with basically only pwd, ls and cd is working just fine ;).

Answer 20

I did chown -R /usr. Somehow, I thought it was a good idea. Subsequently, a puppet script did some bad stuff, and somehow I lost sudo rights.

Our qualified sysadmin wasn't able to recover my sudo rights, and I had to re-image the machine.

Answer 21

I wanted to delete a file and its backup copy (file and file~) on a SuSE system I rarely used and didn't notice that command completion was configured differently. I typed rm fi<tab> and expected the cursor to be after the last letter. The system ignored ~ files and inserted a space after the file name.

So I wanted to type

rm file*

and I did

rm file *

Now I am always checking what the suggested completion is.

Answer 22

A while ago, I needed to do some extensive configuration on one of my machines, that mostly involved editing bunch of files in /etc. I decided to be really careful about it, so I created an etc directory in my $HOME, copied the files that needed editing in there, spent couple hours doing the edits. I carefully checked all the files, made sure that all the edits were exactly the way they were supposed to be, logged in as root and copied the edited files back to /etc. By then, it was very late at night. Still as root, I decided to clean up, and instead of rm -rf etc, I typed in rm -rf /etc. I did not get much sleep that night.

Answer 23

I was on the phone with a colleague who was out at a customer site. She was working on their systems and I was telling her stroke for stroke what to type, she misheard me when I told her rm -rf .??* and typed rm -rf .?*. But she wasn't in the directory she had told me, she was in the root directory. Wiped not only the dot files, but the entire OS.

Back when I was "learning as a sysadmin", I was writing my own adduser script (didn't have it on early SysV). A shell error in the script (cat /etc/passwd; echo ...) > /etc/passwd which of course wiped the passwd file and then, by accident, I hit Ctrl-D to exit the su shell. Had to go in to the office at 1am Saturday morning to get the boot diskettes.