Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

We're working on a small embedded Linux system (2.6.35-ish) with a smallish internal NAND device for the OS and applications (250-500Meg) and an SD card with 8Gb SDHC SD cards for data.

The unit's power can be cut at any time.

The system must store data to SD cards. This data is pretty important... it's the whole purpose of the system. The systems are usually entirely disconnected from any network in remote locations and data is retrieved via sneakernet every 4-8 weeks.

Currently, we've simply got VFAT on the SD cards. That was mainly so the first test clients could easily copy data off manually onto their Win7 laptops.

However, I'm now worried that it's only a matter of time until a power outage at the wrong time causes data loss.

What's the best way to configure such a system to prevent data loss? JFFS2 sounds like what I would want in terms of how it writes data (and the performance needs are not high at all), but it sounds fairly kludgy to use block2mtd, etc. I'm also not sure how the card's wear leveling will interact with it.

What's the best way to do this?

EDIT

I'm now thinking of leaving the filesystem VFAT and allocating day-sized files at a time, filled with 0xFF, that should greatly limit the exposure to power cycle failures. I could then only append records within those precreated blocks, and hopefully the SD cards aren't so stupid that they'd erase/wear level writes to 0xFF regions.

I can use noatime, but is there a VFAT nomtime equivalent to prevent writes to the modified time field? I'd need some way to prevent any metadata updates at all until a new day's file is created.

EDIT 2

Someone on the electronics stack exchange reminded me there's also ECC data on NAND, so there's no way to prevent the need for an erase.

So, would JFFS2 via block2mtd be appropriate in this situation?

EDIT 3

It's worse than I thought. The SD cards I have will erase the data blocks even if you write the exact same contents to disk. The eraseblocks are 64KB, and that's too large to entirely delay writes for. I'll store up to 128KB of data in NAND flash (which I can control the write behavior of), in a kind of journal, and then write 128KB blocks to a 128KB-aligned file in a VFAT partition on the SD card (in case other SD cards have 128KB eraseblocks).

share|improve this question
    
"hopefully the SD cards aren't so stupid…" <--- ROFLOL. Not likely! –  derobert Dec 10 '11 at 4:06
    
Until you find a full solution to the problem you have, use the sync command after each write to the SD card, it would write the bits imidiatly after you have changed/created them without storing them in the RAM so your changes atleast be on the card and wouldn't gone by the power loss. –  Hanan N. Dec 11 '11 at 7:43
    
sync would probably make matters worse, since it increases the fraction of time that the metadata is mid-update. –  Ben Voigt Oct 11 '13 at 20:31

4 Answers 4

Well, the way you can fix this is to fix the "power can be cut at any time" problem. Is it impossible to add even a minute of battery power?

Alternatively, maybe you could use two SD cards. Write the data to one card, sync, write to the other. Each block of your data would need a checksum and block number, but then even with some pretty unlucky power failures, one of the cards should be right.

Your basic problem is going to be wear leveling on the SD cards, which AFAIK depends on the card vendor (and maybe even the batch, they can change it whenever). It probably doesn't handle power outage correctly. And depending on what it does, that may not just mean corrupting the block you're writing to.

  1. Assume trivially small card—3 (flash) blocks. Block 1 has received more writes than 2 or 3. I'll call the physical blocks by number, and logical blocks A, B, C by letter. Right now, A=1, B=2, C=3.
  2. You issue a write to block A . SD card is like aha! we need wear leveling here, else block 1 is going to wear out before 2 and 3. It decides to swap block 1 and 2.
  3. It reads block 1 into RAM position i (on the SD card, not system RAM). It updates the part you wanted to change.
  4. It reads block 2 into RAM position ii
  5. It erases block 1
  6. It writes RAM position ii to block 1.
  7. It updates the mapping table to say B=1
  8. It erases block 2.
  9. It writes RAM position i to block 2.
  10. It updates the mapping table to say A=2

Of course, "updates the mapping table" isn't always trivial. And the order of 5–10 could be different (if they all complete, it doesn't matter, well the erases have to come before writes, of course). But a power failure happens, you could wind up with not only A corrupted (as you expect) but B as well. Or, if power failure happens during a mapping update, who knows what kind of corruption that'll cause.

share|improve this answer
1  
These units must live in relatively harsh environments for many years and when installed they'll get shipped around to various countries that we'd rather not have to qualify batteries for. We'd probably drop MMC/SD and build our own NAND-flash solution before using a battery. –  darron Dec 10 '11 at 4:53
    
Well, in our case the "fix the 'power can be cut at any time'" solution boils down to "stop truck drivers from falling asleep behind the wheel and driving into our devices." "A truck crashed into it" is actually about the most common mode of failure. –  SF. Aug 2 '13 at 7:47
    
A minute of battery power shouldn't be needed. The amount of power needed to safely unmount an SD card ought to be well within the range that a capacitor can store. –  Ben Voigt Oct 11 '13 at 20:33

Something similar was discussed in electronics.stackexchange.com: How do I protect SD card against unexpected power failures?

A sideways answer which works in tandem with software solutions is to look at the hardware (there was a question on ESE about this too, but I can't find it now; it wasn't strictly about SD cards, just about devices losing power and how to detect this and act on it).

The short story is: you may not have battery power, but your power supply has some pretty large capacitors to smooth the supply. Basically, power doesn't just go out. Voltage tapers off. There's probably a brown-out protection IC/circuit which asserts the RESET signal on your embedded system when the voltage drops below a certain point. PC motherboards have those too, and they responds to the ‘POWEROK’ signal from the PSU. What this means is that, when power goes out, the computer will be forcefully halted a few milliseconds before the voltage drops below safe levels. During this time, peripherals like SD cards are still powered up, but there are no more transactions coming from the computer.

It's very likely that an SD card has enough time to complete any pending transactions including wear levelling, before its power goes out. Improving your power supply with a large enough capacitor or using one near the SD card may help ensure this, but you can always experiment with your platform as is. It's quite likely that it retains power for enough time.

If the hardware aspect of the problem is a non-issue, you can solve the software-only ones. derobert's ides of using two cards for redundancy isn't bad, and using a standard filesystem like VFAT runs less of a risk of confusing the card's wear levelling algorithms.

Anyway, it could well be that you won't have that much of an issue. Assuming a block on your card can survive 100 writes (conservative — but do try to get good quality cards!), and using 8GB cards, you'll have written 800GB by the time the first block dies (statistically speaking, of course).

share|improve this answer
    
The question was started because I was already getting massive SD card corruption on power off events. Pretty often, actually. Maybe 1 in 20 power failure events were catastrophic, and possibly 1 in 4 did at least SOME damage. I eventually changed to storing a day's worth of data on internal NAND flash, and copying to SD at midnight (a 1 second or so operation). I'm looking to improve things in the future. I've already got 400uF worth of caps on the rail... not enough, apparently... perhaps the brownout is not being handled properly. –  darron Jun 26 '12 at 21:38
    
That's a pretty high incidence rate! Time to get the oscilloscope probes and see this in action, methinks. Although it's likely you can work around it in software, the best way in terms of power consumption is to ensure you don't have hardware glitches. Maybe you could hedge your bets and ask on electronics.stackexchange.com as well? –  Alexios Jun 26 '12 at 22:03
    
@darron, what solution did you end up with for your SD card storage problem? Are you still writing to NANDFlash then copying over once a day? I have a design with SD card as the main RFS (no separate NANDFlash) and am seeing data corruption problems, both with and without sudden power fail conditions. –  fred basset Jun 6 at 3:15

For Data Security in an environment with the possibility of power cuts and overall data security you should conider even more points.

USE NO MLC cells for storage, only SLC have a data retention time which is sufficient. Then those SLC cards can have smart firmware, some cannot under any state be corrupted by power loss. They recognize the power cut by measuring and secure that the last block is written completly.

Those cards are more expensive and a bit slower then the MLC cells. See vendors like swissbit for cards.

share|improve this answer

We had a problem with our SD, ext2 root file-system being corrupted on unexpected power fails. First of all, we make the system run with a read-only root mount. Since we needed some writeable storage (but we were not data logging) we set up a second partition as writeable. To minimise the FS damage on unexpected power fail we made this an ext3 partition, even though this will cause at least twice as many physical writes to the card. This combination (but I admit our second partition writes are infrequent compared with a data logger) seems to run without problem. So far. (Systems installed for about 30 months in professional facilities)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.