What are your favorite *painful* Unix moments

Question

How about sharing your favorite lessons learned moments?

Answer 1

When I first started working as a user consultant for the university I was attending, I was given limited sudo rights to help students who had lost/forgotten their passwords. sudo passwd <username> was my new friend. An hour after my orientation, my curiosity got the better of me and I typed in sudo passwd and stared in horror at the prompt for a new password. I was a bit scared to ^C my way out of it, thinking (erroniously, it turns out) that I might leave the account in question in a transient state, so I entered a password and immediately walked upstairs to the hallowed 2nd floor domain of the campus SuperUser and asked if he would like to know the root password of the main system.

Answer 2

Back in the mid to late 90s, a friend of mine and I were discussing the folly of rm -rf * and at what point a Linux box would go belly up. We got into statically linked versus dynamically linked libraries and I posited that the system could live quite well without /lib and then proceeded to rename it on my workstation. Bad things happened, but we were left with several open console windows with which to try and fix the damage (shutdown wasn't an option anymore). None of the editors would run. It is amazing the esoteric uses you can find for the echo command.

Answer 3

Trying to get the Xwindows driver for my Nvidia card working when Fedora initially released the Nouveau driver. I had downloaded the Nvidia source to compile and install myself as I had many times in the past, but this release, I could just not get it to work. There were quite a few steps to find in the Fedora Forums to completely disable the Nouveau driver, and get the Nvidia driver working. Quite painful to say the least.

Answer 4

Not really my moment, but someone else's.

Back when I worked at a nuclear sciences research facility we used to run a number of SunOS, Ultrix and Linux computers and researchers had to share the CPU on those machines. As individual research groups got their own research grants they purchased their own computers, mostly SparcStations and they did the system administration themselves.

SunOS used to ship with the OpenView desktop and a nice file manager, this is what it looked like:

Most of our researchers were running as root, and more than once we had to reinstall their operating systems because someone had decided to tidy-up the root directory and moved /bin, /etc, /tmp and everything else that cluttered the view into either the Trashcan or some subfolder.

Other users chose to tidy up the /bin directory and remove any command they did not know.

The lucky ones had back ups, most had purchased a tape drive, but did not have a tradition of running backups themselves.

Answer 5

Makefile:

clean:
    @rm -f * .o

Which, of course, makes make clean wipe your source code instead of just object files.

Lesson: use version control.

Answer 6

I deleted /etc and then recovered it. I don't think I learned my lesson... I've had to recover from a deleted /bin too. Seems to happen when I've been working with a chroot.

Answer 7

Had a friend run :() { :|:&}; : on a remote server where we didn't have console access to. Couldn't reboot it, completely frozen, production server.

Broken down (by request) to make it a bit more readable.

:() # Define ':' as a function. Every time we say ':' execute the following code block
{ # Start of code block
    : # Call ':' again. 
    | # Pipe output to...
    : # Another ':' 
    & # Disown process. 
    # All on one line this would read :|:&, 
} # End of code block
; # End definition of ':' as a function
: # Call ':'

It might be easier to look at it as

bomb() { bomb|bomb& }; bomb

Answer 8

A long time ago, I was installing MkLinux on my Mac, and I wanted to replace the file that governed command processing (not the shell, something more basic, don't remember quite what anymore). The instructions said to do mv x y, so I decided to be cautious and started with rm y. The intention was to mv x y afterwards, but of course that didn't work. I reinstalled.

Answer 9

Not that painful... But a fun little moment:

I've mistyped ls as sl and found out that the sysadmin had something installed for such case.

(already available in Debian, Ubuntu, Gentoo,... repositories)

Answer 10

When my University decided to switch the wireless network to use proprietary Cisco LEAP authentication...

Started a very long battle that ended well enough. Wrote up documentation for others who wanted to run Linux and have access to the internet. Six months later they decided to add PEAP support as well. face slap

It is my favorite because I won. I got it to work.

Answer 11

vi and Caps-Lock vs. /etc/passwd

1. Connect to an old Solaris box using an old serial terminal that doesn't refresh the screen correctly.
2. su -
3. vi /etc/passwd. There is no vipw, and "we're just making minor edits" anyways.
4. Hit Caps-Lock key and don't notice.
5. Hit j a couple times to scroll down. Ignore the fact that you actually just typed J ("Join"), which combines this line with the next line. The serial terminal screen was not refreshing correctly, so you didn't see that you just combined the first 5 lines into one Loooooong line, thereby corrupting the first 5 users ('root', 'daemon', etc).
6. Finish your OTHER intended edits to the file, way down at the bottom.
7. Save file.
8. Log out.

I did this once. Amazingly, the system remained functional for months. Cronjobs ran fine, no errors stood out in the logfiles.

We didn't notice this problem until we rebooted the system months later and couldn't log in at the console. ps showed a bunch of jobs owned by UID '0' not by user 'root'.

You could not log in as root, nor run su or su -, and there was no sudo on this box. There was no floppy drive, the CD-ROM was busted and no USB ports (so no external CD-ROM). Single user mode did not work, because you need to type in the password for root, and that comes from /etc/passwd.

Answer 12

While in my 2nd year of studying computer science we were given a homework assignment to write a program in C that would spawn a number of subprocesses with fork and make them communicate with pipes in a "circle" and figure out which one should be the "leader".

We were still quite noobs back then and most of peple didn't have any Linux machines, so we worked on our accounts on our faculty's main server (which was hosting official site and staff accounts and sites as well). Most of the people wrote forkbombs at some stage of trying to do the homework. Over half of my group got to the abusers file. That was the highest load on that server in a looong time :)

Answer 13

I had two drives installed at one point and had the root filesystem of the second drive mounted in a directory within /mnt. I was in that directory and tried to delete var but ended up typing rm -rf /var instead. Some instinct seemed to kick in that said var must be preceded with a slash!

When I realised what I'd done I immediately hit Ctrl-C but it was too late. My rpm database had long since left the building. I spent ages getting everything back to normal.

Now for the painful part.

I go back into that directory in /mnt to resume what I'd been doing. What do I type? Well, let's just say that instinct kicked in again.

At least I was able to restore the system a lot quicker the second time ;)

Answer 14

Surprised nobody else has mentioned this one yet:

rm -rf .*

(While attempting to remove all hidden files and subdirectories, completely forgetting that it will recurse into . and ..)

Answer 15

I meant well, I really did. Trying to chmod recursively a directory and ended up swapping ./ with /.

As root of course, because only with root can true pain (and thus enlightenment) be achieved.

Answer 16

chown nobody:nobody /*

As root of course....

Answer 17

rm -f * ~

and

rm -rf ${DIR}/

when DIR was not set!

Answer 18

Or another experience, how to feel really stupid in a few easy steps that don't seem all that stupid individually.

Step one: establish an account for the kid, in case he wants to use a Linux box. Give it a trivial password, since after all this is a home system and isn't exposed to the net.

Step two: allow time to elapse, so you don't remember step one.

Step three: open the SSH port in the firewall (actually the NAT on the router) in order to ssh in. After all, my accounts have pretty good passwords, and it isn't like there's anything tremendously valuable.

Step four: get notification from ISP that there's some sort of DOS activity going to a Swedish site. Assume it's probably the Windows boxes, and examine and harden them.

Step five: get notification from ISP that it's still going on. Ask for some detail, get IP address of Swedish site, fire up Wireshark, find which box the attack is coming from.

Step six: clean up Linux box, feeling stupid. Find the login came from a Romanian address. Remove accounts without good passwords.

Answer 19

On a Debian installation back in 1999. 14 floppy disks for the basic installation.

I tried to get xfree86 working. But X didn't start. I had to find out the settings of my graphics card (memory, horizontal and vertical refresh rate), which was completely undocumented. It turned out superprobe found out the correct amount of internal graphics card memory (1024 kB).

But it took me nearly a week to find out that the resolution setting (1024x768) didn't work. I had to switch it to 640x480 until the graphics card finally worked (at 1024x768... buuuuuug....).

I tried to get the serial port mouse to work on COM1. So I tried to get the mouse to work. Reading a book (back then I had no usable high-speed internet), I tried with

/dev/ttys0

And it didn't work and didn't work. It took me nearly another week to find out this was because I needed to type the S is uppercase, not lowercase...

/dev/ttyS0

It was about then when I finally realized what 'case-sensitive' really means.

Answer 20

A simple halt recognizing some seconds later that I'm not on a local shell and having no possibility to power on the production server again.

Lessons learned? The prompt of the machine now looks like

[ --> root <-- @kompost:/home/echox] #

with some nice red markup ;-)

Answer 21

I wiped the partition table of my main drive by accident, thinking I was working on another drive.

With scrollback, careful use of df, memory, and luck I was able to recreate it exactly, rewrite it, reboot, and hope... And it worked.

Answer 22

I remember trying to send a SysRq key sequence to a remote machine...

...but it was captured by the local one.

Answer 23

A company that I used to work for had its product running on SCO. I was doing some debugging about applications getting very slow on our demo server and at the same time there was a bunch of customers being given a demo/lecture about upcoming new features.

So, I ran the application that used to get stuck, did my stuff on it to verify the root cause, but since it was still "stuck", I tried to kill it:

pkill -9 mytestapplication

What I did learn was that pkill doesn't do exactly the same on SCO as it does on linux =)

... It basically kills everything the user has access to, and with root... that's everything =)

Answer 24

Back in the day you had to do erase the first 512 bytes of a partition to properly format FAT drives from Linux. This is done using the dd command.

dd if=/dev/zero of=/dev/hda1 bs=512 count=1

Except the FAT partition was /dev/hdb1

I didn't realize what had happened until after I rebooted. Luckily I was able to recover it by re-installing Lilo, or something.

Answer 25

Twenty minutes ago, I was painstakingly recreating a complex directory structure from files I had on my other PC. I decided to run du to see if it was near completion. Given the size, I knew it would take few minutes, so went to get some coffee.

On my return I noticed, to my severe dismay, that instead of running

du -hs /path/to/important/folder

I had absent-mindedly run

rm -r  /path/to/important/folder

Answer 26

I was once developing a device driver for Unix. It had a pointer problem and during testing it started to write off the end of an array in kernel memory. I was slow to spot this and didn't hit the reset button immediately. The driver had scribbled all over the disk buffer cache which was then flushed to disk before I hit reset. A lot of the blocks were inodes and directories, and I ended up with a totally trashed filesystem. I think 6000 orphaned files were put into lost+found before I gave up and reinstalled. Fortunately, this was only a test system, not my workstation with all my files on it.

Answer 27

This happened to me last year. I was removing some files from the server using a temporary variable:

rm -rf ${prefix}*

Guess what? The variable $prefix was not defined!
You can imagine the disaster... it resulted in some very critical files deleted.

I almost broke the Control-C and ran to the CPU to remove the network cable!!

Hahaha I'm sure someone had already done this...

Answer 28

Mine was chmod -R 777 /: after that I couldn't figure out how to restore the permission on the whole filesystem and then I reinstalled the OS.

Never did that again (and still don't know how to restore such a situation).

Answer 29

I was a lab assistant for a Linux class. One of the students called me over because she could no longer su - because she was getting permission denied. OK, she's misremembered/mistyped the password. Reboot into single-user mode and reset. What?! su STILL doesn't work?! It MUST bow to my will! So I reboot into single user mode to find out what she did. I realized that she ran chmod -R 777 /var/www/html/drupal-6.19 /

Note the space between the directory name and the final slash.

After a few minutes of "I really don't want to have her reinstall, so what is this doing and how.", I managed to find that /bin/su now had file permissions of 777. That can also be read as file permissions of 0777, which removes the setuid bit from /bin/su. A quick chmod u+s /bin/su and I was a hero.

Answer 30

I was curious if chmod 000 / would work.

Well, flawlessly. A few minutes later I was searching for a rescue CD.