My old gentoo laptop seem to have an issue with udev probably due to me updating software using
emerge, upgrading udev+kernel, getting some messages "do these things after upgrading udev/kernel" and thinking "I'll look into the screen output, read those messages and take care of those things later before ever rebooting the system". Unfortunately, being a bit in a hurry, I did, of course, forget to review those messages, shut down the machine, moved it physically to it's now location at home and started up, only to be greeted with a failure to find
Now I have managed to work around that (described in length below) by manually creating
/dev/hda3 and then making
init continue the system boot but I need some advice on how to proceed on fixing this once and for all (so I don't have to repeat the procedure belown on every boot).
(If anyone can point to somewhere Gentoo machine where I can find all post-install messages for the packages I have installed, then I can find those instructions related to udev and the kernel which I should have read.)
Relevant software versions:
Gentoo-hardened-kernel 2.6.36-r6 (preferable) and 2.6.28-r9 (used for 21 months before shutdown). The cumbersome manual startup process described below has been verified to work for both kernel versions. The udev package is udev-151-r4.
When booting up everything seems to be fine (the machine start's the kernel, it mounts /proc, /sys and /dev, starts udevd and populates /dev based on uevents, processes the uevents, mounts /dev/pts) until it comes to the step of 'Checking root filesystem'.
There it emits
Failed to open the device '/dev/hda3': No such file or directory
Then it asks me for root password (or Ctrl-D to continue).
I enter that, and in the shell,
mount tells me that
rootfs is mounted on and that
/dev/root also is mounted on
mount complains that
/etc/mtab is not writable (e.g read/only filesystem). All of this makes sense to me, given that the bootup is not completed.
ls /dev reveals that there are no
hda3 devices, which normally should be there (and which I would expect udev to create). Also worth mentioning,
/dev contains no
sda3 devices either, so we are not seeing the issue of an updated udev or kernel changing it's convention of what to call different disk devices. Neither is there any
/dev/disk directory where disk devices could "hide".
So my next, potentially dirty, step is to manually create
/dev (I had a sneak peek at one of my other gentoo servers to figure out major and minor numbers as well as appropriate permissions and group membership):
mknod /dev/hda b 3 0
mknod /dev/hda1 b 3 1
mknod /dev/hda2 b 3 2
mknod /dev/hda3 b 3 3
chmod 660 /dev/hda*
chgrp disk /dev/hda*
Unfortunately, doing Ctrl-D or writing
exit here will not continue the interrupted boot sequence, but instead it will initiate a reboot (which brings the machine up to the state of
/dev/hda not found), so that path to solving this issue won't help, unfortunately.
Another workaround I tried (which failed) where to press 'I' (as well as trying 'i') as soon as
init started and began to run the various init scripts (like mounting /proc, /sys and so on) but I never manage to enter interactive boot mode before the boot sequence reaches the (failed) attempt to check /dev/hda3.
Instead I continue with using my
/dev/hda3 device to mount my OS filesystem:
mount -o remount -o rw /dev/hda3 /
So, now I have write access to the file system of my machine which gives me some options regarding troubleshooting this situation: amending configuration files, starting various init-scripts and subsystems etc.
One thing that does not work is to change run-level though. The reason is that
init 3 fails with the error message
init: /dev/initctl: No such file or directory. Again I do a sneak peek on my other Gentoo server and finds out that
/dev/initctl is a pipe with permissions
600 and belonging to
root:root, so I recreate that:
mknod /dev/initctl p
chmod 600 /dev/initctl
init 3 fails, but a little differently; it hangs for a while and then gives up with the message
init: timeout opening/writing control channel /dev/initctl. This makes sense as the original
init process (with process id 1) does not have this freshly created
/dev/initctl opened for reading.
Now, reading the
init man page I realize that sending
SIGUSR will make
init close and reopen
/dev/initctl. Exactly what I need, so I execute the command
kill -l to get a list of all signals and their number (in which I see that
SIGUSR1 has the number 10) then I issue the command
kill -10 1
/dev/initctl, then I retry entering run level 3:
init tries to enter runlevel 3 and executes a large number of scripts. Unfortunately, all those scripts fails with
ERROR: cannot run syslog-ng until sysinit completes. So, I reboot the system (which now actually works as expected as I've made
init listen to
/dev/initctl: I logon as
root and then issues
reboot), repeats the above steps (except for remounting
/dev/hda3 as writable) up to and including the point of sending
init. Now I am trying to make
init resume the boot sequence but in a more gentle way, by making it reread the
Nothing seem to have happened. So, I'll investigate
/etc/inittab where I find an entry for a runlevel which seem to called
sysinit. I take the risk and rerun that:
init complains with a usage message. Reading
/etc/inittab again, I see that the
sysinit entry makes a call to
/sbin/rc with the argument
sysinit. So, I decide to try that:
Now the system retries to boot up a number of services and succeeds! Not only that, when re-running the init scripts for mounting
/dev there is a check made to see if they already are mounted (a testament of how valuable it is to do sanity checks and error checks in code and act accordingly when running into exceptional circumstances). Happy with this, I decide to also run the commands for the
bootwait, as that one as well is missing a runlevel letter or digit.
Again, a number of init scripts are started, mainly for networking. No unexpected errors are reported, so I'm happy to retry getting to runlevel 3:
Once the initscripts have finished, the machine is up and I can logon as root (and, in theory, fix the root cause of this mess)!