Corruption-proof SD card filesystem for embedded Linux?

Question

Recently we had a rather unpleasant situation with our customer - Raspberry Pi based "kiosk" used to display remote sensing data (nothing more fancy than a kiosk mode browser displaying a self-updating webpage from the data-collection server) failed to boot due to filesystem corruption. Ext4, Manual fsck required, the system will be a part of tomorrow's important presentation, service required immediately. Of course we can't require the customer to shut down the system nicely when switching it off for the night; the system must simply withstand such mistreatment.

I'd like to avoid such situations in the future, and I'd like to move the OS to a filesystem that would prevent this. There's a bunch of filesystems intended for MTD devices, where getting them to run on SD card (a standard block device) requires some serious hoop-jumping. There are also some other filesystems (journalling etc) that boast good resistance against corruption. I still need to see some reasonable comparison of their pros and cons.

Which filesystem available in Linux would provide best resistance against corruption on unexpected power failures and not require jumping through impossible hoops like yaffs2 in order to install to SD.

Wear-balancing is a plus, but not a requirement - SD cards usually have their own mechanisms, if less than perfect, though the system should be "gentle for flash" (systems like NTFS can murder an SD card within a month).

Answer 1

The best resistance against corruption on a single SD card would be offered by BTRFS in RAID1 mode with automatic scrub run every predefined period of time.

The benefits:

1. retaining ability to RW to the filesystem
2. modern, fully featured filesystem with very useful options for an RPi, like transparent compression and snapshots
3. designed with flash memory in mind (among other things)

---

Here is how to do it:

I run my RaspberryPi on ArchARM linux and my card is in the SD reader, so modify those instructions accordingly for other distros and /dev interfaces.

Here is an example partition layout:

/dev/mmcblk0p1: fat32 boot partition
/dev/mmcblk0p2: to be used as btrfs partition
/dev/mmcblk0p3: to be used as btrfs partition (mirrored with the above)
/dev/mmcblk0p4 (optional): swap

To get btrfs into RAID1, you create the filesystem like so:

mkfs.btrfs -m raid1 -d raid1 /dev/mmcblk0p2 /dev/mmcblk0p3

Then you rsync -aAXv to it your previously backed up system.

To get it to boot from BTRFS in raid1, you need to modify initramfs. Therefore, you need to do the following while you still have your system running on your old filesystem.

Raspberry does not normally use mkinitcpio so you must install it. Then, you need to add “btrfs” to MODULES array in mkinitcpio.conf and recreate initramfs with

mkinitcpio -g /boot/initrd -k YOUR_KERNEL_VERSION

To know what to type instead of YOUR_KERNEL_VERSION, run

ls /lib/modules

If you update the kernel, you MUST recreate initramfs BEFORE you reboot.

Then, you need to modify RPi’s boot files.

In cmdline.txt, you need to have

root=/dev/mmcblk0p2 initrd=0x01f00000 rootfstype=btrfs

and in config.txt, you need to add

initramfs initrd 0x01f00000

Once you’ve done all that and successfully booted into your btrfs RAID1 system, the only thing left is to set up periodic scrub (every 3-7 days) either with systemd timer (preferred), or cron (dcron) like so:

btrfs scrub start /

It will run on your filesystem comparing checksums of all the files and fixing them (replacing with the correct copy) if it finds any corruption.

The combination of BTRFS RAID1, single medium and Raspberry Pi make this pretty arcane stuff. It took some time and work to put all the pieces together, but here it is.

Answer 2

I would go another way and would just use a read-only filesystem. I never get my raspberry pi stable enough when using a read-write root filesystem on the sdcard. You can either just boot your root via kernel cmdline (ro) or use a initramfs with piggyback including your complete system.

Both are possible to create with my homemade build system OpenADK. (http://www.openadk.org)

Answer 3

Well flash storage is more desirable than magnetic storage, for multiple reasons, but for this application I'll say mainly because there is no moving parts. That being said, I don't think there is a 'corruption proof' filesystem out there, but there are some robust filesystems (ext4 being one) out there, as well as some tactics to help mitigate corruption.

RAM Disk

If the RPi's image does not have to change, and it sounds like it doesn't, if nothing will try to (or should be trying to) write to the disk, then try using a root filesystem created to be unpacked into RAM. The idea here is that you have a compressed root filesystem on boot up that gets unpacked into RAM. All changes occur to the RAM disk, so there is effectively zero writing to the SD Card, only reading at boot. this should cut down on the reads/writes to your drive, preserving the life of it. This is similar to what's done when you boot linux from a CD, and is one of the first things that happens when linux boots.

Answer 4

Well, the problem you're having here is that using a "modern" filesystem such as the ext* is likely to wear your SD Card down ; from my experience that happens within a year, or the next year if you take the higher end.

The problem being that modern filesystems are always moving blocks around to prevent fragmentation of data. Which is a good thing on spinning disks, where you want to have all your data collated when loading up it to the cache. The downside is that it's doing more writes that can't be cached as the tidying up is being handled when there isn't much I/O happening.

It's also happening when you do handle a lot of logging, which you may want to do when debugging your embedded device. The logging writes are the worst kind of writes, because it's lots of tiny writes happening regularly, which generates a lot of fragmentation.

As you say your system is also handling sensor's data, it is very likely that you store them on the flash as they come. And they're as bad as the log data.

I went into the same problem you're running into, and here are my conclusions. I tried to look for SD Cards that would be sold as being "more robust", i.e. being able to handle more writes than the others, but I found no benchmark on the market that focuses on that, unlike the benchmarks on the SSD. As they all focus on speed only, it's impossible to know the number of writes per memory block, and the technology used in the SDCard.

You want to look for an SD Card using SLC memory, typically only available from suppliers like Mouser or Digikey. This technology is true "industrial" grade SD card, and has a lifetime orders of magnitude more than any consumer grade MLC/TLC SD Card on the market

Though, I've noticed the "industrial" grade sandisks had a longer lifespan that no-names. Which isn't surprising, when you pay more, you get more.

But in the end, with intensive logging enabled, I found no SD Card having a lifespan longer than a couple of years, one year being where the most death occurs.

The solution I came up with are @BigHomie's and @wbx' solutions: use a Read Only extX filesystem (as journalizing is no longer needed, you can even fallback to the good old ext2). And if you want to keep logs within the session or write temporary files, you can always use a RAMDISK.

There exists tutorials and scripts only that helps populate the ramdisk with data from within the read only parts so you can edit them for the session.

N.B.: my experience has been using Angstrom Linux on a Beaglebone, amongst a trial run of 20 sensor devices. That system's logging was very verbose, using systemd's journal system.

Answer 5

Linux offers many filesystems. ext4 is the one I have more confidence in. When in doubt, ext4 should be used for any partition that will be mounted read-write.

The ext2 filesystem is much more fragile. It is a perfectly good filesystem for systems that are able to mount it read-only, or unmount it correctly. But corruption is extremely likely with a power failure on ext2.

The other option could be considering jfs even though jfs filesystem is not reliable in some Linux versions. Corruption is less likely with jfs than with ext4. Jfs is also has a quick mount time and file system check time.

Answer 6

DO NOT USE A JOURNALLED FILESYSTEM on an SD, miniSD, microSD, or "USB Stick" memory device.

Journalled filesystems increase the number of writes to the device, which makes it wear out faster, which make potential corruption events (reaching the wear-out limit) more frequent. Journaling filesystems are designed for spinning magnetic media which can be written to essentially an infinite number of times, not media which wears out after 10k ~ 100k writes to the same area (typically 1M or larger block size), even if only a single bit is changed in the entire block. It has to do with the fact that writing to NAND-based flash memory adds charges to MOS transistors, which, by design, have characteristics of capacitors, and unfortunately, what makes them retain data with power off ALSO results in undrainable permanent charge accumulation every time a 0 is written to the bit (the slowly accumulating charge can't even be reversed by writing 1 to the same bit location)

This categorically rules out ext4, xfs, and jfs.

Don't use F2FS or UBIFS. F2FS is meant for raw MTDs with no wear leveling and UBIFS is meant specifically for UBI devices with no wear-leveling.

USB sticks and the various sizes of *SD cards all have wear leveling because they're "consumer" products made for people whose computer expertise starts and ends at using their cell phones and copying pictures from one office computer to another.

BTRFS appears to currently be the least damaging to those NAND-based devices which have their own internal wear-levelling such as USB sticks and *SD cards, by limiting the rate that it writes to the storage device, that being said, BTRFS still isn't entirely stable.

I don't know where "SSD drives" manufactured with SATA interfaces fall in this realm.

Me? I leave only the absolute essentials on my RPi's microSD card, and I get as big a card as I can find (currently 256G size), and I moved the swap and as much of the filesystem as possible onto a couple partitions on a 1 TB 2.5" laptop drive, connected through a powered USB3.0 hub (to keep from burning out the controller chip for my RPi's USB 3.0 port by offloading the current consumption to a separate power supply).

If the RPi4 would let me boot directly off of a USB port (Correction, see below at the end of my response) like RPi3 does, I would have the entire OS installed on the disk like I did with RPI3 (but unfortunately, RPi3 only has USB 2.0, which is a painfully slow way to access mass storage, and RPI3's have only 4GB, even though the CPU cores are aarm64 architecture, eliminating a significant portion of the benefit of 64-bit address space.)

This is not definitive, it's merely the best my research turns up at the present (Feb 2022).

See here for F2FS and what type of solid state memory it's designed for and what hardware it shouldn't be used on: http://www.linux-mtd.infradead.org/faq/general.html

See here for UBIFS and what type of solid state memory it's designed for and what it shouldn't be used on: http://www.linux-mtd.infradead.org/faq/ubifs.html

CORRECTION: rpi-imager now has a method for changing your RPi 4 and RPi 400 to boot from USB (and also boot from network): https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb