Twice in 6 months - on three Ext4 disks - one of them a Seagate that is about a year old - I have had catastrophic corruption of the filesystem. What seems to happen is that in many directories the list of directory entries is truncated, leaving only "." and "..", and much of the metadata structures get overwritten with straight 0xFFs. That is, a block will consist solely of the the byte 0xFF repeated 4096 times. I use Ubuntu Mate, 18.04.3 LTS, 4.15.0-72-generic #81-Ubuntu SMP.
In the first instance about 6 months ago, I took two disks out of a machine that had suffered a power supply failure, mounted them in another machine without funning fsck on them. I did not write any data to the disks, but since I neglected to mount them read-only, metadata was probably written. Eventually I ran fsck on them both, and the above corruption happened. Specifically, the superblock, all backup superblocks, all copies of the GDTs, and large parts of the inode tables were overwritten with 0xFF bytes. By writing code to scan the disk for inode records outside of the proper tables and for directory files, I was able to cobble much of the filesystems back together. I estimate I recovered around half of the data on one disk, and 75% on the other. Well, I was pretty careless to mount the drives RW without running fsck on them, even though I did not overtly write data. I took this as a hard lesson to not be so careless. Curiously enough, I did find some of the missing metadata in the wrong places on the disk. This is how I was able to recover the superblock, and quite a few inode table entries. It appeared as if the system in its confusion wrote them to several data areas on the disk. Filtering them by checksums, I found 5-10 copies of some of the items in these scans.
However, only a few days ago, similar corruption happened again (perhaps not as severe, I hope). I had a data disk with a single Ext4 partition on it, default options, in a machine that was having power-supply problems. The machine kept spontaneously turning off. The disk was a 4TB Seagate that was only a year old. I finally got a replacement power supply, installed it, booted it up, and found that the data disk was corrupted. I made several (5 ?) attempts to boot, each time getting dropped into emergency/safe mode. The annoying splash screen hid from me what was going on during the boot attempts. I finally realized that the boot was failing because my data disk was listed in fstab, but was not mountable. Removing it from fstab, the system booted ok. Looking at the disk, I found it corrupted. The superblock - exactly all of block 0 - was all 0xFFs. However, the backup suberblocks were fine, as were the GDTs. Of the inode entries in use, around 0.5% of them was also overwritten with straight 0xFFs - including the first inode block (inodes 1-11) (that is, 99.5% of them look fine). The 0xFF overwrite was always an integer number of blocks, on block boundaries. As I have begun to examine the filesystem in more detail, I have found nearly all of the inode entries of directory files were overwritten with the 0xFFs - a whole block at a time. So, although its only half a percent of the inodes in use, its the most inconvenient half of a percent to loose. Also, several of the higher-level directory files looked as though all of the entries (except ".", "..") had been deleted. That is, the rec_len field of the ".." entry was extended to reach to the end of the block, and a subsequent block of the directory file had the inode number of the first entry set to zero.
So, it looks as if when I tried to boot, fsck was automatically run. For reasons I do not understand, it deleted all files in several directories, and then overwrote key metadata blocks with straight 0xFFs. It did this automatically - without asking for my approval for this "fix".
I have several questions:
1) Could this be due to the filesystem journal getting corrupted, and then fsck going nuts trying reconcile? If not, what could cause this?
2) Is there some reason that Ext4 with journaling is actually more vulnerable to data loss - through this failure mode - than say, an older ext2 or ext3 filesystem?
3) How can I configure the next data disk filesystem to make as certain as possible this never happens again? I plan to recover what I can of the most recent data disk, and then set up an Ext4 filesystem with checksumming enabled. Also, I heard that there was an option to enable checksumming of the journal - is that advisable? Reliability is more important to me than performance - but I also don't have the funds to set up a large raid array. Is there a better filesystem than Ext4 for my purpose?
4) Can I set an option in Ubuntu Mate to limit what fsck will attempt to automatically do to "fix" my filesystem (on boot, or otherwise)? Would it be better for me to disable journaling? I don't want fsck automatically deleting files! In fact, I am seriously considering just deleting fsck and writing my own repair code - but there has got to be an easier way.
I googled around online to see if other people have had this happen to them, and can not find any other cases. If this is so common of a failure mode to happen to me three times in six months, surely it must have affected many other people out there. However, I can not find any other descriptions of this problem on the web.