How to troubleshoot udev not creating /dev/hda3 on gentoo boot?

Question

My old gentoo laptop seem to have an issue with udev probably due to me updating software using emerge, upgrading udev+kernel, getting some messages "do these things after upgrading udev/kernel" and thinking "I'll look into the screen output, read those messages and take care of those things later before ever rebooting the system". Unfortunately, being a bit in a hurry, I did, of course, forget to review those messages, shut down the machine, moved it physically to it's now location at home and started up, only to be greeted with a failure to find /dev/hda3.

Now I have managed to work around that (described in length below) by manually creating /dev/hda3 and then making init continue the system boot but I need some advice on how to proceed on fixing this once and for all (so I don't have to repeat the procedure belown on every boot).

(If anyone can point to somewhere Gentoo machine where I can find all post-install messages for the packages I have installed, then I can find those instructions related to udev and the kernel which I should have read.)

Relevant software versions:

Gentoo-hardened-kernel 2.6.36-r6 (preferable) and 2.6.28-r9 (used for 21 months before shutdown). The cumbersome manual startup process described below has been verified to work for both kernel versions. The udev package is udev-151-r4.

When booting up everything seems to be fine (the machine start's the kernel, it mounts /proc, /sys and /dev, starts udevd and populates /dev based on uevents, processes the uevents, mounts /dev/pts) until it comes to the step of 'Checking root filesystem'.

There it emits

Failed to open the device '/dev/hda3': No such file or directory

Then it asks me for root password (or Ctrl-D to continue).

I enter that, and in the shell, mount tells me that rootfs is mounted on and that /dev/root also is mounted on /. Also, mount complains that /etc/mtab is not writable (e.g read/only filesystem). All of this makes sense to me, given that the bootup is not completed.

Doing ls /dev reveals that there are no hda, hda1, hda2 or hda3 devices, which normally should be there (and which I would expect udev to create). Also worth mentioning, /dev contains no sda, sda1, sda2 or sda3 devices either, so we are not seeing the issue of an updated udev or kernel changing it's convention of what to call different disk devices. Neither is there any /dev/disk directory where disk devices could "hide".

So my next, potentially dirty, step is to manually create hda, hda1, hda2 and hda3 under /dev (I had a sneak peek at one of my other gentoo servers to figure out major and minor numbers as well as appropriate permissions and group membership):

mknod /dev/hda b 3 0
mknod /dev/hda1 b 3 1
mknod /dev/hda2 b 3 2
mknod /dev/hda3 b 3 3
chmod 660 /dev/hda*
chgrp disk /dev/hda*

Unfortunately, doing Ctrl-D or writing exit here will not continue the interrupted boot sequence, but instead it will initiate a reboot (which brings the machine up to the state of /dev/hda not found), so that path to solving this issue won't help, unfortunately.

Another workaround I tried (which failed) where to press 'I' (as well as trying 'i') as soon as init started and began to run the various init scripts (like mounting /proc, /sys and so on) but I never manage to enter interactive boot mode before the boot sequence reaches the (failed) attempt to check /dev/hda3.

Instead I continue with using my /dev/hda3 device to mount my OS filesystem:
mount -o remount -o rw /dev/hda3 /

So, now I have write access to the file system of my machine which gives me some options regarding troubleshooting this situation: amending configuration files, starting various init-scripts and subsystems etc.

One thing that does not work is to change run-level though. The reason is that init 3 fails with the error message init: /dev/initctl: No such file or directory. Again I do a sneak peek on my other Gentoo server and finds out that /dev/initctl is a pipe with permissions 600 and belonging to root:root, so I recreate that:

mknod /dev/initctl p chmod 600 /dev/initctl

Now init 3 fails, but a little differently; it hangs for a while and then gives up with the message init: timeout opening/writing control channel /dev/initctl. This makes sense as the original init process (with process id 1) does not have this freshly created /dev/initctl opened for reading.

Now, reading the init man page I realize that sending SIGUSR will make init close and reopen /dev/initctl. Exactly what I need, so I execute the command kill -l to get a list of all signals and their number (in which I see that SIGUSR1 has the number 10) then I issue the command
kill -10 1
to make init reopen /dev/initctl, then I retry entering run level 3:
init 3

Now init tries to enter runlevel 3 and executes a large number of scripts. Unfortunately, all those scripts fails with ERROR: cannot run syslog-ng until sysinit completes. So, I reboot the system (which now actually works as expected as I've made init listen to /dev/initctl: I logon as root and then issues reboot), repeats the above steps (except for remounting /dev/hda3 as writable) up to and including the point of sending SIGUSR1 to init. Now I am trying to make init resume the boot sequence but in a more gentle way, by making it reread the /etc/inittab file:

init q

Nothing seem to have happened. So, I'll investigate /etc/inittab where I find an entry for a runlevel which seem to called sysinit. I take the risk and rerun that:

init sysinit

This time init complains with a usage message. Reading /etc/inittab again, I see that the sysinit entry makes a call to /sbin/rc with the argument sysinit. So, I decide to try that:

/sbin/rc sysinit

Now the system retries to boot up a number of services and succeeds! Not only that, when re-running the init scripts for mounting /proc, /sys and /dev there is a check made to see if they already are mounted (a testament of how valuable it is to do sanity checks and error checks in code and act accordingly when running into exceptional circumstances). Happy with this, I decide to also run the commands for the /etc/inittab entry bootwait, as that one as well is missing a runlevel letter or digit.

/sbin/rc boot

Again, a number of init scripts are started, mainly for networking. No unexpected errors are reported, so I'm happy to retry getting to runlevel 3:

init 3

Once the initscripts have finished, the machine is up and I can logon as root (and, in theory, fix the root cause of this mess)!

Answer 1

I recompiled the kernel making sure that neither of CONFIG_SYSFS_DEPRECATED nor CONFIG_SYSFS_DEPRECATED_V2 were set.

However, after a few retries with CONFIG_IDE not set (and a kernel which failed to find any harddrives), it turned out I had to CONFIG_IDE set as well as some more kernel config options found 'under' the deprecated ATA/ATAPI support. Again, I had a sneak peak at that other server (which have properly working /dev/hda detection) and added the missing ATA/ATAPI related kernel options. Which options I actually added I did not note, but here is the list of all the options activated underATA/ATAPI: CONFIG_IDE_GD
CONFIG_IDE_GD_ATA
CONFIG_IDE_PROC_FS
CONFIG_IDE_GENERIC
CONFIG_IDE_PCIBUS_ORDER
CONFIG_BLK_DEV_GENERIC
CONFIG_BLK_DEV_PIIX

I also disabled the support for serial ATA and paralell ATA.

So now I have a kernel which boots up and successfully finds my /dev/hda3 device.

I do know this is a bad practice, to rely on the deprecated ATA/ATAPI kernel support and not use the more modern SATA/PATA kernel support. Eventually I have to migrate to SATA/PATA instead. But for now, I'm happy (and a bit more knowledgeable about my Linux system).

(And my apologies if I post a question and then answers it and picks my own answer for being the accepted one. Hopefully this answer/question still is a useful contribution to U&L)