Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Recently we had a rather unpleasant situation with our customer - Raspberry Pi based "kiosk" used to display remote sensing data (nothing more fancy than a kiosk mode browser displaying a self-updating webpage from the data-collection server) failed to boot due to filesystem corruption. Ext4, Manual fsck required, the system will be a part of tomorrow's important presentation, service required immediately. Of course we can't require the customer to shut down the system nicely when switching it off for the night; the system must simply withstand such mistreatment.

I'd like to avoid such situations in the future, and I'd like to move the OS to a filesystem that would prevent this. There's a bunch of filesystems intended for MTD devices, where getting them to run on SD card (a standard block device) requires some serious hoop-jumping. There are also some other filesystems (journalling etc) that boast good resistance against corruption. I still need to see some reasonable comparison of their pros and cons.

Which filesystem available in Linux would provide best resistance against corruption on unexpected power failures and not require jumping through impossible hoops like yaffs2 in order to install to SD.

Wear-balancing is a plus, but not a requirement - SD cards usually have their own mechanisms, if less than perfect, though the system should be "gentle for flash" (systems like NTFS can murder an SD card within a month).

share|improve this question

migrated from embedded.stackexchange.com Jun 9 '14 at 20:48

1  
Personally, I'd go the other way, and work on a safe shutdown on poweroff, probably using a cap to provide enough oomph to run a shutdown. –  Scott Seidman May 21 '14 at 10:38
    
I'd love to see someone design the module that provides just enough power for a clean shutdown, along with the needed system support to heed the warning and actually shut down. It seems like it ought to be a sensible companion to the Pi, BeagleBone, and other tiny Linux machines, but it doesn't seem to exist as a product marked to users of those machines. –  RBerteig May 22 '14 at 1:11
    
@ScottSeidman: This being RPi, quite power-hungry - think 800mA at 5V for 15s. Not quite a capacitor thing unless you invest in a whole battery of supercaps. –  SF. May 22 '14 at 4:34
    
@RBerteig: A box with a rechargeable battery, proper electronics to charge, stabilize output voltage (possibly step-up from the battery's own), send shutdown signal, cut output power after shutdown is completed, shut down self until input power is restored - sure it's all doable but if you don't manufacture that in bulk, it about doubles the cost of the RPi (although in case of that kiosk, the TV set is 10 times more expensive...) –  SF. May 22 '14 at 4:44
1  
@SF. - Note that there's two issues with a robust against powerfail file system. The first is that the FS itself be robust, the second is that the underlying hardware not lie about flushing data to disk. I know that spinning disks have taken to lying in recent years to boost their apparent performance, you'll want to make sure that your SD cards aren't doing the same. –  Michael Kohne May 22 '14 at 19:18

5 Answers 5

up vote 2 down vote accepted

The best resistance against corruption on a single SD card would be offered by BTRFS in RAID1 mode with automatic scrub run every predefined period of time.

The benefits:

  1. retaining ability to RW to the filesystem
  2. modern, fully featured filesystem with very useful options for an RPi, like transparent compression and snapshots
  3. designed with flash memory in mind (among other things)

Here is how to do it:

I run my RaspberryPi on ArchARM linux and my card is in the SD reader, so modify those instructions accordingly for other distros and /dev interfaces.

Here is an example partition layout:

/dev/mmcblk0p1: fat32 boot partition
/dev/mmcblk0p2: to be used as btrfs partition
/dev/mmcblk0p3: to be used as btrfs partition (mirrored with the above)
/dev/mmcblk0p4 (optional): swap

To get btrfs into RAID1, you create the filesystem like so:

mkfs.btrfs -m raid1 -d raid1 /dev/mmcblk0p2 /dev/mmcblk0p3

Then you rsync -aAXv to it your previously backed up system.

To get it to boot from BTRFS in raid1, you need to modify initramfs. Therefore, you need to do the following while you still have your system running on your old filesystem.

Raspberry does not normally use mkinitcpio so you must install it. Then, you need to add “btrfs” to MODULES array in mkinitcpio.conf and recreate initramfs with

mkinitcpio -g /boot/initrd -k YOUR_KERNEL_VERSION

To know what to type instead of YOUR_KERNEL_VERSION, run

ls /lib/modules

If you update the kernel, you MUST recreate initramfs BEFORE you reboot.

Then, you need to modify RPi’s boot files.

In cmdline.txt, you need to have

root=/dev/mmcblk0p2 initrd=0x01f00000 rootfstype=btrfs

and in config.txt, you need to add

initramfs initrd 0x01f00000

Once you’ve done all that and successfully booted into your btrfs RAID1 system, the only thing left is to set up periodic scrub (every 3-7 days) either with systemd timer (preferred), or cron (dcron) like so:

btrfs scrub start /

It will run on your filesystem comparing checksums of all the files and fixing them (replacing with the correct copy) if it finds any corruption.

The combination of BTRFS RAID1, single medium and Raspberry Pi make this pretty arcane stuff. It took some time and work to put all the pieces together, but here it is.

share|improve this answer
    
Should I add the 'scrub' after each boot-up as well? –  SF. Feb 26 at 14:10
    
@SF. No, it is not necessary. Periodic scrub every X days is sufficient. Preferably during hours of least usage. –  lockheed Feb 27 at 14:52

I would go another way and would just use a read-only filesystem. I never get my raspberry pi stable enough when using a read-write root filesystem on the sdcard. You can either just boot your root via kernel cmdline (ro) or use a initramfs with piggyback including your complete system.

Both are possible to create with my homemade build system OpenADK. (http://www.openadk.org)

share|improve this answer

Well flash storage is more desirable than magnetic storage, for multiple reasons, but for this application I'll say mainly because there is no moving parts. That being said, I don't think there is a 'corruption proof' filesystem out there, but there are some robust filesystems (ext4 being one) out there, as well as some tactics to help mitigate corruption.

RAM Disk

If the RPi's image does not have to change, and it sounds like it doesn't, if nothing will try to (or should be trying to) write to the disk, then try using a root filesystem created to be unpacked into RAM. The idea here is that you have a compressed root filesystem on boot up that gets unpacked into RAM. All changes occur to the RAM disk, so there is effectively zero writing to the SD Card, only reading at boot. this should cut down on the reads/writes to your drive, preserving the life of it. This is similar to what's done when you boot linux from a CD, and is one of the first things that happens when linux boots.

share|improve this answer

Linux offers many filesystems. ext4 is the one I have more confidence in. When in doubt, ext4 should be used for any partition that will be mounted read-write.

The ext2 filesystem is much more fragile. It is a perfectly good filesystem for systems that are able to mount it read-only, or unmount it correctly. But corruption is extremely likely with a power failure on ext2.

The other option could be considering jfs even though jfs filesystem is not reliable in some Linux versions. Corruption is less likely with jfs than with ext4. Jfs is also has a quick mount time and file system check time.

share|improve this answer

Well, the problem you're having here is that using a "modern" filesystem such as the ext* is likely to wear your SD Card down ; from my experience that happens within a year, or the next year if you take the higher end.

The problem being that modern filesystems are always moving blocks around to prevent fragmentation of data. Which is a good thing on spinning disks, where you want to have all your data collated when loading up it to the cache. The downside is that it's doing more writes that can't be cached as the tidying up is being handled when there isn't much I/O happening.

It's also happening when you do handle a lot of logging, which you may want to do when debugging your embedded device. The logging writes are the worst kind of writes, because it's lots of tiny writes happening regularly, which generates a lot of fragmentation.

As you say your system is also handling sensor's data, it is very likely that you store them on the flash as they come. And they're as bad as the log data.

I went into the same problem you're running into, and here are my conclusions. I tried to look for SD Cards that would be sold as being "more robust", i.e. being able to handle more writes than the others, but I found no benchmark on the market that focuses on that, unlike the benchmarks on the SSD. As they all focus on speed only, it's impossible to know the number of writes per memory block, and the technology used in the SDCard.

Though, I've noticed the "industrial" grade sandisks had a longer lifespan that no-names. Which isn't surprising, when you pay more, you get more.

But in the end, with intensive logging enabled, I found no SD Card having a lifespan longer than a couple of years, one year being where the most death occurs.

The solution I came up with are @BigHomie's and @wbx' solutions: use a Read Only extX filesystem (as journalizing is no longer needed, you can even fallback to the good old ext2). And if you want to keep logs within the session or write temporary files, you can always use a RAMDISK.

There exists tutorials and scripts only that helps populate the ramdisk with data from within the read only parts so you can edit them for the session.

N.B.: my experience has been using Angstrom Linux on a Beaglebone, amongst a trial run of 20 sensor devices. That system's logging was very verbose, using systemd's journal system.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.