Can Ext4 journal corruption catastrophically corrupt the whole filesystem by deleting files & overwriting with FFs?

Question

Twice in 6 months - on three Ext4 disks - one of them a Seagate that is about a year old - I have had catastrophic corruption of the filesystem. What seems to happen is that in many directories the list of directory entries is truncated, leaving only "." and "..", and much of the metadata structures get overwritten with straight 0xFFs. That is, a block will consist solely of the the byte 0xFF repeated 4096 times. I use Ubuntu Mate, 18.04.3 LTS, 4.15.0-72-generic #81-Ubuntu SMP.

In the first instance about 6 months ago, I took two disks out of a machine that had suffered a power supply failure, mounted them in another machine without funning fsck on them. I did not write any data to the disks, but since I neglected to mount them read-only, metadata was probably written. Eventually I ran fsck on them both, and the above corruption happened. Specifically, the superblock, all backup superblocks, all copies of the GDTs, and large parts of the inode tables were overwritten with 0xFF bytes. By writing code to scan the disk for inode records outside of the proper tables and for directory files, I was able to cobble much of the filesystems back together. I estimate I recovered around half of the data on one disk, and 75% on the other. Well, I was pretty careless to mount the drives RW without running fsck on them, even though I did not overtly write data. I took this as a hard lesson to not be so careless. Curiously enough, I did find some of the missing metadata in the wrong places on the disk. This is how I was able to recover the superblock, and quite a few inode table entries. It appeared as if the system in its confusion wrote them to several data areas on the disk. Filtering them by checksums, I found 5-10 copies of some of the items in these scans.

However, only a few days ago, similar corruption happened again (perhaps not as severe, I hope). I had a data disk with a single Ext4 partition on it, default options, in a machine that was having power-supply problems. The machine kept spontaneously turning off. The disk was a 4TB Seagate that was only a year old. I finally got a replacement power supply, installed it, booted it up, and found that the data disk was corrupted. I made several (5 ?) attempts to boot, each time getting dropped into emergency/safe mode. The annoying splash screen hid from me what was going on during the boot attempts. I finally realized that the boot was failing because my data disk was listed in fstab, but was not mountable. Removing it from fstab, the system booted ok. Looking at the disk, I found it corrupted. The superblock - exactly all of block 0 - was all 0xFFs. However, the backup suberblocks were fine, as were the GDTs. Of the inode entries in use, around 0.5% of them was also overwritten with straight 0xFFs - including the first inode block (inodes 1-11) (that is, 99.5% of them look fine). The 0xFF overwrite was always an integer number of blocks, on block boundaries. As I have begun to examine the filesystem in more detail, I have found nearly all of the inode entries of directory files were overwritten with the 0xFFs - a whole block at a time. So, although its only half a percent of the inodes in use, its the most inconvenient half of a percent to loose. Also, several of the higher-level directory files looked as though all of the entries (except ".", "..") had been deleted. That is, the rec_len field of the ".." entry was extended to reach to the end of the block, and a subsequent block of the directory file had the inode number of the first entry set to zero.

So, it looks as if when I tried to boot, fsck was automatically run. For reasons I do not understand, it deleted all files in several directories, and then overwrote key metadata blocks with straight 0xFFs. It did this automatically - without asking for my approval for this "fix".

I have several questions:

1. Could this be due to the filesystem journal getting corrupted, and then fsck going nuts trying reconcile? If not, what could cause this?

2. Is there some reason that Ext4 with journaling is actually more vulnerable to data loss - through this failure mode - than say, an older ext2 or ext3 filesystem?

3. How can I configure the next data disk filesystem to make as certain as possible this never happens again? I plan to recover what I can of the most recent data disk, and then set up an Ext4 filesystem with checksumming enabled. Also, I heard that there was an option to enable checksumming of the journal - is that advisable? Reliability is more important to me than performance - but I also don't have the funds to set up a large raid array. Is there a better filesystem than Ext4 for my purpose?

4. Can I set an option in Ubuntu Mate to limit what fsck will attempt to automatically do to "fix" my filesystem (on boot, or otherwise)? Would it be better for me to disable journaling? I don't want fsck automatically deleting files! In fact, I am seriously considering just deleting fsck and writing my own repair code - but there has got to be an easier way.

I googled around online to see if other people have had this happen to them, and can not find any other cases. If this is so common of a failure mode to happen to me three times in six months, surely it must have affected many other people out there. However, I can not find any other descriptions of this problem on the web.

Answer 1

please check out the block_validity and journal_checksum options. without those, ext4 has much less safeties.

you can find some historic stuff on this at https://ext4.wiki.kernel.org/index.php/Ext4_Metadata_Checksums

Answer 2

Unfortunately, I'm not an expert on ext4 beyond 'light geeky' use.

To avoid these things happening again, you can take some steps, such as:

1. Never save on power supply units (PSUs) by going for the cheapest ones, which may or may not fail. There's just no telling.

2. Optionally, if you do get the funds available, you could buy an Uninterrupted Power Supply (UPS) to put between your wall socket and computer's PSU. It will provide battery backup in case of a power outage and, additionally, will provide an extra protection against surges, spikes and other strange things that can happen on the grid. I can tell by experience that it is a good investment. You may want to go for a 400W (or more if your computer and other devices draw more power) UPS just to keep your computer running for a few minutes so that you can shut it down gracefully, if the need arises, instead of seeing it all go black and quiet instantly. A UPS might keep your PSU happier as well.

3. Create a live rescue system. Knoppix is a distribution especially around for that reason. I recently experimented a bit with MX Linux, which allows you to customize the live CD environment on a USB device (like a 16GB flash drive) with persistence, which means that settings and changes are saved. So you could remove a lot of unneeded software and add a whole bunch of tools that may ever be useful for recovery of some sort. Remaster the live system with some extra space for small backups (such as undo files) and you're good to go. If something like this happens, just run the live system so that it won't auto-check any filesystems on the internal disk(s) etc. A live system can also be helpful for accessing online resources to find information or other assistance if needed.

4. Use a boot loader with menu (GRUB2 is usually a good choice), that will wait for a few seconds before getting on with the default boot entry. It will allow you to do something else, before going straight into a potential automated frenzy, as you pretty much described it.