What would happen if a hard drive failed while the Linux kernel was running?

Question

I've always been curious of what would happen if a hard drive failed while the Linux kernel was running. Since the kernel image itself is loaded completely into RAM, I don't imagine there would be any kernel failures. Obviously any software that attempted to access the disk would fail.

Is there any chance that the system would continue running indefinitely? If not, what would cause it to panic (or at least become inoperable)?

Please consider each of the following scenarios:

• The kernel may or may not have the swapfile enabled.
• No attempts to load kernel modules are made after the failure.
• Does it make a difference if certain parts of the filesystem are on different disks?

Answer 1

Hardware failures always run some risk of crashing the Kernel since those code paths generally have had much less testing, but normally, a failed hard drive should not crash the Kernel. What exactly happens depends on the nature of the failure. Perhaps only certain sectors are now unreadable rendering parts of the /home partition unreadable, the system will still be runnable for a sysadmin to analyze the problem. If the root filesystem becomes unusable, the system is pretty much dead regardless of a Kernel crash as even a simple shell won't be available. If a swap partition becomes unavailable, programs that are using swap will segment fault when it comes time to read in any swapped out data. If the hard drive that crashed is simply extra storage, it may have little affect besides some filesystems becoming unreadable.

It can also depend on what kind of errors the hard drive is throwing. I've seen a drive effectively disappear and besides the file systems disappearing, everything ran ok. I've also seen a hard drive continually hanging the system and throwing errors after a long timeout causing the whole system performance to degrade. If using a layer like MD running RAID1/4/5, a severe error will normally just cause the Kernel to mark the disk as failed, and it will ignore it relying on the remaining drives to keep the system running.

Answer 2

On my PowerEdge 2500, when I first got it, the PERC (hardware RAID) controller's firmware was not at the latest revision. The effect of this is that the root disk would just suddenly disappear and would no longer be accessible (very similar to if it was a removable drive and it was just suddenly disconnected).

I couldn't load any new programs, programs that were loaded kept running, but with errors if they attempted to write to the disk. Still had the bash prompt I had logged into, network continued to function. Was surprisingly not as catastrophic as I would have expected.

I think this is a "clean" failure though, because whatever driver was responsible for reading/writing to the PERC seemed to be rejecting everything immediately with an error (forget the exact one but it was a SCSI sense error). It would be much worse if the drive wasn't responding, responding slowly, or writes appeared to be working OK but really weren't.

Answer 3

I actually had a drive fail on a running system, no X though. No logins were possible because getty was not available. I tried shelling out of a running app, but no shell was available and the app was then unusable. That is when the Aiieeee message came to my attention and told the story.