8

I am running a small uClibc and busybox based embedded system on an x86 device. I am using an initramfs but I also mounting a custom ext3 directory on a compact flash device in IDE mode that I am using to store persistent measurement logging data created by a custom written c++ application. I chose the ext3 file system as it is recommended for safety against power loss when using CF drives in IDE mode in a couple of books I have read (Building Embedded Linux Systems by Karim Yaghmour and Embedded Linux Primer by Christopher Hallinan). This is particularly important and the data is critical.

However, due to some of the comments in my previous question Confusion with how to restore corrupt ext3 files if power outage occurs during a file write it would appear that in fact this file system does not offer the guarantee of safety against data corruption due to power loss. So I would like to know if

  1. Is ext3 actually the best choice for this setup?
  2. Does power loss during a disc write operation only corrupt the portion of data I am appending to the file periodically or can it corrupt the entire file?
  3. Is data that is not being written at the point of power loss completely safe? In particular, is there any risk that my initramfs.cpio file can become corrupt also?
  4. Is there any method I can use in my application code to protect the data (i.e. creating an extra partition and writing my data to mirror images so that there are always 2 copies) - speed is not a real issue for my application so expensive copying operations are acceptable.

I have seen and read the answers to this related question: Do journaling filesystems guarantee against corruption after a power failure?, but it doesn't quite cover some of the things that are confusing me.

I realise that I am asking a lot of questions but it seems that despite reading a lot of material I have had a fundamental failure to understand the risks to my data in the event of power loss.

Improve this question

4 Answers 4

Reset to default
11

As with all things pertaining to security, there aren't any guarantees, but you also need to balance risk (and cost) against probability. From experience (and I've been running dozens of *nix boxen since the dark ages), I've never really had significant power-caused filesystem corruption.

Some of these machines were even running on non-journalled filesystems (ufs and ext2 usually). Some of them were embedded, and a few were mobile phones like the Nokia N900 — so a good power supply wasn't at all guaranteed.

It's not that filesystem corruption can't happen, it's just that the probability of it happening is low enough that it shouldn't worry you. Still, no reason not to hedge your bets.

In answer to your literal questions:

  1. At least the first book you referenced was written before ext4 — when the author suggests using ext3, they're really saying ‘don't use unstable or non-journalled filesystems like ext2’). Try ext4, it's quite mature, and has some decent options for non-spinning disks which may extend the life expectancy of your flash device.
  2. Chances are it would lose you the last block or two, not the entire file. With a journalled filesystem, this will be about the only loss. There are failure scenarios where I could see random data sprayed across the file, but they seem about as likely as a micrometeorite smashing right through your embedded device.
  3. See 2. Nothing is 100.00% safe.

  4. If you have a second IDE channel, stick a second CF card in there and grab a backup of the filesystem periodically. There are a few ways to do this: rsync, cp dump, dd, even using the md(4) (software RAID) device (you add the second drive occasionally, let it sync, then remove it — if both devices are live all the time, they run the same risk of filesystem corruption). If you use LVM, you can even grab snapshots. For a data collection embedded device, I'd just use am ad hoc solution which mounts the second filesystem, copies over the data log, the immediately unmounts it. If you're worried about the device having a good boot image, stick a second copy of the boot manager and all necessary boot images on the second device and configure the computer to boot from either CF card.

    I wouldn't trust a second copy on the same device because storage devices fail more often than stable filesystems. Much more often, in my experience so far (at work, there was a bitter half-joke about the uncannily high chances of Friday afternoon disk failures. It was almost a weekly event for a while). Whether the disk is spinning or not, it can fail. So keep your eggs in two baskets if you can, and you'll protect your data better.

    If the data is particularly sensitive, I'd pay regular visits to the device, swap the backup CF for a fresh one and reboot, letting it fsck all its filesystems for good measure.

Improve this answer
3
  • +1, however replication suffers from the same problems as the primary copy - if you start syncing two devices (be it through RAID or higher level utility) and the power goes out (while there is constant appending to the data), you'll get garbage again. What might help is having RAID1, from time to time physically changing one of the devices and making an off-line backup form the one removed. You'll need to freeze the FS before removing it though, to make sure it's consistent (i.e. make snapshots). XFS is one of the file systems that have support for this.
    – peterph
    Mar 25, 2013 at 11:59
  • Indeed. Like I wrote, there aren't any guarantees. Any time you're writing data, you could have corruption. People at electronics.stackexchange.com have been playing around with supercapacitors and brown-out detection where the embedded system gets a notification the power's out, and still gets enough juice to abort writes. Maybe. :) It's all a matter of how likely you think the potential danger is, and how much money/effort you want to spend to remove the issue at hand (and start considering the next one).
    – Alexios
    Mar 25, 2013 at 13:29
  • Thanks for this answer. This clarifies things for me considerably.
    – mathematician1975
    Mar 26, 2013 at 10:09
4

It seems to me that what a filesystem implementation can achieve in the case of sudden power loss is limited -- after all, it is actually interfacing with the hardware, so what happens between the time it sends data/instructions to the hardware and when it gets a response is out of its control. If there were a filesystem that could circumvent this issue you would have heard of it.

Because of that, a strategy to protect critical data will benefit most from decisions made on a hardware level, eg, by using an uninterruptible power supply. Probably this is not so feasible in your situation.

You've said performance is not really a big issue, so make judicious use of fsync().

Does power loss during a disc write operation only corrupt the portion of data I am appending to the file periodically or can it corrupt the entire file?

I've been been using extN filesystems personally and on low-medium traffic internet servers for years, and like Alexios I haven't seen much corruption due to power failure (although to be fair, the servers have UPS and I can't recall one of them actually going down that way). A much more serious issue is corruption from hardware failure, which different filesystems may (again) be more and less capable of dealing with the problem, but (again) this is fundamentally beyond their control and they can't prevent it.

I have occasionally seen files lost, or truncated to zero size. I presume there is a good chance these would be recoverable somehow; this was not necessary for me as they were backed up. Most of the time if there is anything wrong at all fsck seems to deal with it.

Is data that is not being written at the point of power loss completely safe? In particular, is there any risk that my initramfs.cpio file can become corrupt also?

I think that risk is really really low from just a power failure, excepting the kind of corruption flash storage may be subject to due to the power surge that can accompany power failures -- which I have no experience with, but hopefully you have thought about and researched this.

Is there any method I can use in my application code to protect the data?

Worth repeating the point about fsync(). C++/iostream objects don't have a method for this (::flush and ::sync are not fsync), but all you need is a file descriptor.

Improve this answer
2
  • Thankyou for this answer it is very helpful also. I am mounting the partition that gets written to via sync option in the /etc/fstab file as I understand that this forces the write to happen synchronously. I am assuming that this means that when my file write code returns, then the data has been physically written to the disc. I understood that mounting with sync essentially does the same as calling fsync(my_filedescriptor) after a write. Is my understanding of this correct?
    – mathematician1975
    Mar 26, 2013 at 10:17
  • @mathematician1975 I would presume so, this is not something I have researched. IMO, as long as it is not somehow inconvenient, throwing fsync() in at points you think is appropriate won't hurt anyway, and makes the system more robust (eg, if the device is casually mounted without sync set, etc).
    – goldilocks
    Mar 26, 2013 at 15:33
1

ZFS is definitely a file system protected against corruption by design and possibly the only one. However, I'm not sure about the availability of ZFS implementations (either fuse based or native) for uClinux based platforms.

Improve this answer
0

There is at least one commercial file system that does a tremendous job making sure that the file-system very nearly cannot be corrupted due to power failures and that the only data you risk loosing is data that was being added as the power went out.

The down-side is that it is very expensive, on the up-side they offer great support. Due to the expense, its really only an option for high stakes and/or high volume products. Like critical embedded equipment in e.g. oil and gas production that need to ensure system integrity within "uncertain" operating conditions (e.g. frequent power outages, etc.).

Check out DataLight (company) and/or product "Reliance NITRO". (Reliance is their legacy and safe but not very effective solution, superseded by Reliance NITRO). Even if you don't have money to use this system, they have some pretty good articles discussing how their system works, why it is more reliable than e.g. ext3 and ext4.

My apologies if this read like an ad, just wanted to point out options.

Improve this answer
1
  • Hi and welcome to the site. If you are going to suggest products, please i) provide a link to the product in question; ii) explain why it is better than alternatives (you just claim it does a tremendous job but don't explain why it is better than anything else); iii) if you are affiliated with the company making this, you need to make that explicit or be accused of spamming (not saying that you are, just a heads up).
    – terdon ♦
    Oct 30, 2014 at 19:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .