4

We have occasional power outages in our environment which seems to cause data corruption on our Ubuntu machines with ext4 filesystems.

To my understanding ext4's default is to use data=ordered

Which is described as "All data are forced directly out to the main file system prior to its metadata being committed to the journal."

Does this mean that if there is a power outage, and the operation to write to disk is interrupted that there can be filesystem corruption?

If I want to completely eliminate filesystem corruption due to power outages I'd guess I would use data=journaled, are there any negative impacts to this other than a performance hit?

Bonus: How do I change the journaling type on my filesystem from data=ordered to another. I'm guessing I'd need to make modifications to the journal but I'm not quite sure how or in what order to perform these operations.

It's just getting really annoying that Ubuntu (initramfs) doesn't have any filesystem recovery utilities so any way we can get to prevent us from having to pop in a live cd is great.

My /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
#                
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=9cd71f51-53bb-44c7-affa-14293e59d596 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=5568cee1-a50b-4409-ad67-cdc5bfb592a3 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0

OS version

-bash-4.0# uname -a
Linux LG-F3-19 2.6.31-14-server #48-Ubuntu SMP Fri Oct 16 15:07:34 UTC 2009 x86_64 GNU/Linux
-bash-4.0# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 9.10
Release:        9.10
Codename:       karmic

Image of failure: http://imgur.com/odo4iBY

References: https://www.kernel.org/doc/Documentation/filesystems/ext4.txt http://www.ibm.com/developerworks/library/l-journaling-filesystems/

  • Presuming we don't already have it, may I suggest asking a question about how to add fsck, etc. to the initramfs? – derobert Aug 5 '14 at 13:56
  • You might try using a /forcefsck in your root filesystem; this should be a text file containing just -y. That forces fsck -y to be run on the root fs (before it is remounted rw) no matter what. Adds a bit of time to the boot, but should save the hassle of using a live CD if there's an issue. It seems odd that the pattern is your kernel and initramfs are OK, but then there's something else that goes wrong? You haven't actually described the nature of the "corruption" or how you come to this conclusion. – goldilocks Aug 5 '14 at 14:20
  • @goldilocks /forcefsk is literally a file in the root filesystem with a "-y"? How does it know what this file is used for, there must be a config somewhere looking for this file. Last time this happened even when I ran an fsck it would not run because of bad superblocks. So I had to find an alternative and run fsck from liveCD. – rainereality Aug 5 '14 at 17:13
  • Yes, it's just a file with options to fsck. It works on debian and ubuntu systems; although I don't have 9.10 system to check, you should find a reference to it with grep forcefsck /etc/init/mountall.conf. It also gets deleted every boot, so if you want to make it permanent, you need to add something to a start-up service/script to put it back there (you want it there from the beginning in case the power is killed, etc). – goldilocks Aug 6 '14 at 14:38
3

All three data journaling modes should leave the filesystem itself fully intact after a power failure. So it should always mount without errors. The difference is only in the data in your files; data=writeback mode may leave stale data (i.e., what was stored in the disk sectors before the writes your app did). data=ordered and data=journaled should not do this.

Most likely what you're seeing is that I/O barriers aren't working on your setup. First, make sure you're not mounting with barrier=0/nobarrier. That boosts performance, but will cause corruption on power failure.

If I/O barriers are on, it's also possible you're passing through a storage layer that doesn't support them. On older releases, LVM didn't and various mdraid levels didn't. (This was fixed in Linux 2.6.33; so only if you're running Lucid still.)

Finally, it's possible your disks are telling lies. Disks have write caches. Especially with NCQ, they're supposed to only tell the OS they've written data when they've actually done so, but they've been known to tell the OS its written when its only in the disk's write cache. Increases performance. At least as long as the power stays on. You can try disabling the write cache on the disks, though you'll take a performance hit for this.

Note also that flash-memory disks have a lot of work to do under the hood, and many of them don't handle power failure well. (For example, wear leveling sometimes requires that a full flash block of data be moved. If the power fails in the middle, bad things happen on some flash disks.)

Finally... have you considered an UPS?

  • 1
    +1 For have you considered an [sic] UPS? It seems like you are saying the data loss is a bit unusual; which I agree with. There are of course many anecdotal reports like this on line, but I think they are probably the exception (no one is going to write in when the power fails and everything is fine afterward). Personally, I can't remember seeing any significant corruption after a power fail in ~15 years of using ext filesystems under linux. – goldilocks Aug 5 '14 at 14:15
  • @derobert The system is not using LVM, they are also spinning disks. I do not believe the disks are configured to use write cache. In this environment UPS is not worth it. It's a lab environment – rainereality Aug 5 '14 at 17:11
  • 1
    Unless you've specifically disabled the write cache (e.g., with hdparm -W/sdparm --clear=WCE/etc... often every boot) then the write cache is probably on (it defaults to on). And you can always use cheap, small UPSes in your lab environment. – derobert Aug 5 '14 at 17:14
  • You will have a write cache, most disk drives have them. The only question is does the drive lie about finishing the write. Did you check that barriers are enabled? I had to enable them on my system, don't know why they where disabled (yes faster, a bit, but ). – ctrl-alt-delor Aug 5 '14 at 17:36
  • 1
    @rainereality oh, that error is really clear: your disk has bad blocks. That's a hardware problem. – derobert Aug 5 '14 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.