Recover damaged grub loader on Scientific Linux 6.3

Question

So... there is a notebook far away from me, and I can only sit in front of it in about a month. So I'm searching for a solution - "remotely".

Scientific Linux 6.3 is installed on it, and it worked perfectly. One day the people that use it sent me this picture.

Q: How can I fix the GRUB menu? So that they can use the notebook again?

P.S.: There was an auto-update configured on that machine besides the base install, here is how I implemented it:

yum remove -y PackageKit* yum-autoupdate yum-updateonboot; read; vi /etc/yum-update

---

cat yum-update
#!/bin/bash
# run an update in at least every 172800 seconds [in at least every 48h]

# wait for random time:  0-2700 sec [0-45min]
/bin/sleep $[ ($RANDOM % 2701 ) ]

if [ `stat --format=%Y $0` -le $(( `date +%s` - 172800 )) ]; then 
    # sync time - http://www.pool.ntp.org/zone/hu
        /usr/sbin/ntpdate time.kfki.hu 0.hu.pool.ntp.org; /sbin/hwclock --systohc

    # do updates
        /usr/bin/yum -q -y update
        /usr/bin/yum clean all

    # update last modification time + create log entry
        /bin/touch $0
        /bin/echo "/etc/yum-update OK" | /usr/bin/logger
fi

---

chmod +x /etc/yum-update; crontab -e
52 * * * * bash /etc/yum-update

P.P.S.: They tried to press up/down when the machine booted (hoping to select another kernel) but it didn't help.

Answer 1

It sounds like you've somehow corrupted or lost your /boot/grub/menu.lst or /boot/grub/grub.cfg. The good news is that your kernel is probably still there, and probably still bootable (unless your entire filesystem has corrupted or something).

The GRUB shell is actually pretty functional. You can use it to diagnose and fix your broken configuration, but that's quite hard if you're not physically present. A better option is probably to get to a working kernel, then log in remotely (e.g. via ssh) to check things out.

Assuming you can get your colleagues to run some commands, you should be able to boot them into Linux. They can use tab complete to find the names of the relevant files.

1. Load the kernel:

   grub> kernel /vmlin<TAB>

2. Load the initrd:

   grub> initrd /initrd-<TAB>

   Obviously, make sure the versions of the two selected files match.

3. Boot:

   grub> boot

See this guide for screenshots and more details, including some tips for diagnosing your menu problems.

Once you've got them up and running, you'll need to figure out what went wrong. I won't go into that here though, since there are lots of things that might have happened, and corresponding ways to fix the situation. You'll have to do some detective work to find out.

For example: Someone might have accidently deleted a file in /boot. Your menu.cfg may have become corrupted. A kernel update may have screwed up the menu during a routine run of update-grub or grub-mkconfig. Permissions may have been changed incorrectly. Your menu.cfg may have been changed to point at the wrong root device. Your disk may be in trouble (in which case you have bigger problems). And so on.