42

Recently we had a rather unpleasant situation with our customer - Raspberry Pi based "kiosk" used to display remote sensing data (nothing more fancy than a kiosk mode browser displaying a self-updating webpage from the data-collection server) failed to boot due to filesystem corruption. Ext4, Manual fsck required, the system will be a part of tomorrow's important presentation, service required immediately. Of course we can't require the customer to shut down the system nicely when switching it off for the night; the system must simply withstand such mistreatment.

I'd like to avoid such situations in the future, and I'd like to move the OS to a filesystem that would prevent this. There's a bunch of filesystems intended for MTD devices, where getting them to run on SD card (a standard block device) requires some serious hoop-jumping. There are also some other filesystems (journalling etc) that boast good resistance against corruption. I still need to see some reasonable comparison of their pros and cons.

Which filesystem available in Linux would provide best resistance against corruption on unexpected power failures and not require jumping through impossible hoops like yaffs2 in order to install to SD.

Wear-balancing is a plus, but not a requirement - SD cards usually have their own mechanisms, if less than perfect, though the system should be "gentle for flash" (systems like NTFS can murder an SD card within a month).

12
  • 1
    Personally, I'd go the other way, and work on a safe shutdown on poweroff, probably using a cap to provide enough oomph to run a shutdown. May 21, 2014 at 10:38
  • I'd love to see someone design the module that provides just enough power for a clean shutdown, along with the needed system support to heed the warning and actually shut down. It seems like it ought to be a sensible companion to the Pi, BeagleBone, and other tiny Linux machines, but it doesn't seem to exist as a product marked to users of those machines.
    – RBerteig
    May 22, 2014 at 1:11
  • @ScottSeidman: This being RPi, quite power-hungry - think 800mA at 5V for 15s. Not quite a capacitor thing unless you invest in a whole battery of supercaps.
    – SF.
    May 22, 2014 at 4:34
  • @RBerteig: A box with a rechargeable battery, proper electronics to charge, stabilize output voltage (possibly step-up from the battery's own), send shutdown signal, cut output power after shutdown is completed, shut down self until input power is restored - sure it's all doable but if you don't manufacture that in bulk, it about doubles the cost of the RPi (although in case of that kiosk, the TV set is 10 times more expensive...)
    – SF.
    May 22, 2014 at 4:44
  • 1
    @SF. - Note that there's two issues with a robust against powerfail file system. The first is that the FS itself be robust, the second is that the underlying hardware not lie about flushing data to disk. I know that spinning disks have taken to lying in recent years to boost their apparent performance, you'll want to make sure that your SD cards aren't doing the same. May 22, 2014 at 19:18

6 Answers 6

22

The best resistance against corruption on a single SD card would be offered by BTRFS in RAID1 mode with automatic scrub run every predefined period of time.

The benefits:

  1. retaining ability to RW to the filesystem
  2. modern, fully featured filesystem with very useful options for an RPi, like transparent compression and snapshots
  3. designed with flash memory in mind (among other things)

Here is how to do it:

I run my RaspberryPi on ArchARM linux and my card is in the SD reader, so modify those instructions accordingly for other distros and /dev interfaces.

Here is an example partition layout:

/dev/mmcblk0p1: fat32 boot partition
/dev/mmcblk0p2: to be used as btrfs partition
/dev/mmcblk0p3: to be used as btrfs partition (mirrored with the above)
/dev/mmcblk0p4 (optional): swap

To get btrfs into RAID1, you create the filesystem like so:

mkfs.btrfs -m raid1 -d raid1 /dev/mmcblk0p2 /dev/mmcblk0p3

Then you rsync -aAXv to it your previously backed up system.

To get it to boot from BTRFS in raid1, you need to modify initramfs. Therefore, you need to do the following while you still have your system running on your old filesystem.

Raspberry does not normally use mkinitcpio so you must install it. Then, you need to add “btrfs” to MODULES array in mkinitcpio.conf and recreate initramfs with

mkinitcpio -g /boot/initrd -k YOUR_KERNEL_VERSION

To know what to type instead of YOUR_KERNEL_VERSION, run

ls /lib/modules

If you update the kernel, you MUST recreate initramfs BEFORE you reboot.

Then, you need to modify RPi’s boot files.

In cmdline.txt, you need to have

root=/dev/mmcblk0p2 initrd=0x01f00000 rootfstype=btrfs

and in config.txt, you need to add

initramfs initrd 0x01f00000

Once you’ve done all that and successfully booted into your btrfs RAID1 system, the only thing left is to set up periodic scrub (every 3-7 days) either with systemd timer (preferred), or cron (dcron) like so:

btrfs scrub start /

It will run on your filesystem comparing checksums of all the files and fixing them (replacing with the correct copy) if it finds any corruption.

The combination of BTRFS RAID1, single medium and Raspberry Pi make this pretty arcane stuff. It took some time and work to put all the pieces together, but here it is.

8
  • Should I add the 'scrub' after each boot-up as well?
    – SF.
    Feb 26, 2015 at 14:10
  • @SF. No, it is not necessary. Periodic scrub every X days is sufficient. Preferably during hours of least usage.
    – lockheed
    Feb 27, 2015 at 14:52
  • Sorry, I fail to understand - if I keep a fat /boot partition, do I still need to modify initramfs?
    – Bex
    Aug 5, 2017 at 9:00
  • 2
    @@@ UPDATE: @@@ As of now, one can try to use dup-mode instead of RAID1: "mkfs.btrfs --data dup --metadata dup" but I am not 100% sure it is as resilient as RAID1 on a single drive.
    – lockheed
    Aug 18, 2017 at 18:32
  • 1
    Thanks, this answer gave me most of what I needed to setup an older Pi 2 with BTRFS. I'm using the latest (as of today) ArchLinux ARM build, which has kernel 5.4.51, and it booted just fine without having to modify/recreate the initramfs, which was a nice surprise. Jan 11, 2021 at 20:15
11

I would go another way and would just use a read-only filesystem. I never get my raspberry pi stable enough when using a read-write root filesystem on the sdcard. You can either just boot your root via kernel cmdline (ro) or use a initramfs with piggyback including your complete system.

Both are possible to create with my homemade build system OpenADK. (http://www.openadk.org)

1
9

Well flash storage is more desirable than magnetic storage, for multiple reasons, but for this application I'll say mainly because there is no moving parts. That being said, I don't think there is a 'corruption proof' filesystem out there, but there are some robust filesystems (ext4 being one) out there, as well as some tactics to help mitigate corruption.

RAM Disk

If the RPi's image does not have to change, and it sounds like it doesn't, if nothing will try to (or should be trying to) write to the disk, then try using a root filesystem created to be unpacked into RAM. The idea here is that you have a compressed root filesystem on boot up that gets unpacked into RAM. All changes occur to the RAM disk, so there is effectively zero writing to the SD Card, only reading at boot. this should cut down on the reads/writes to your drive, preserving the life of it. This is similar to what's done when you boot linux from a CD, and is one of the first things that happens when linux boots.

7

Well, the problem you're having here is that using a "modern" filesystem such as the ext* is likely to wear your SD Card down ; from my experience that happens within a year, or the next year if you take the higher end.

The problem being that modern filesystems are always moving blocks around to prevent fragmentation of data. Which is a good thing on spinning disks, where you want to have all your data collated when loading up it to the cache. The downside is that it's doing more writes that can't be cached as the tidying up is being handled when there isn't much I/O happening.

It's also happening when you do handle a lot of logging, which you may want to do when debugging your embedded device. The logging writes are the worst kind of writes, because it's lots of tiny writes happening regularly, which generates a lot of fragmentation.

As you say your system is also handling sensor's data, it is very likely that you store them on the flash as they come. And they're as bad as the log data.

I went into the same problem you're running into, and here are my conclusions. I tried to look for SD Cards that would be sold as being "more robust", i.e. being able to handle more writes than the others, but I found no benchmark on the market that focuses on that, unlike the benchmarks on the SSD. As they all focus on speed only, it's impossible to know the number of writes per memory block, and the technology used in the SDCard.

You want to look for an SD Card using SLC memory, typically only available from suppliers like Mouser or Digikey. This technology is true "industrial" grade SD card, and has a lifetime orders of magnitude more than any consumer grade MLC/TLC SD Card on the market

Though, I've noticed the "industrial" grade sandisks had a longer lifespan that no-names. Which isn't surprising, when you pay more, you get more.

But in the end, with intensive logging enabled, I found no SD Card having a lifespan longer than a couple of years, one year being where the most death occurs.

The solution I came up with are @BigHomie's and @wbx' solutions: use a Read Only extX filesystem (as journalizing is no longer needed, you can even fallback to the good old ext2). And if you want to keep logs within the session or write temporary files, you can always use a RAMDISK.

There exists tutorials and scripts only that helps populate the ramdisk with data from within the read only parts so you can edit them for the session.

N.B.: my experience has been using Angstrom Linux on a Beaglebone, amongst a trial run of 20 sensor devices. That system's logging was very verbose, using systemd's journal system.

1
  • To reduce wear due to logging, a system such as AnySyncDaemon (= keeping logs in a ramdisk periodically synced with the disk) is probably the best
    – Nephanth
    Mar 28, 2020 at 16:56
4

Linux offers many filesystems. ext4 is the one I have more confidence in. When in doubt, ext4 should be used for any partition that will be mounted read-write.

The ext2 filesystem is much more fragile. It is a perfectly good filesystem for systems that are able to mount it read-only, or unmount it correctly. But corruption is extremely likely with a power failure on ext2.

The other option could be considering jfs even though jfs filesystem is not reliable in some Linux versions. Corruption is less likely with jfs than with ext4. Jfs is also has a quick mount time and file system check time.

1
  • 3
    Yes, ext4 is great and I use it for most of my servers. BUT this question is about file systems for root system on sd cards. This is a different environment. Is your advice a general one, or do you really advice to use ext4 cards?
    – guettli
    Feb 8, 2017 at 5:57
-1

DO NOT USE A JOURNALLED FILESYSTEM on an SD, miniSD, microSD, or "USB Stick" memory device.

Journalled filesystems increase the number of writes to the device, which makes it wear out faster, which make potential corruption events (reaching the wear-out limit) more frequent. Journaling filesystems are designed for spinning magnetic media which can be written to essentially an infinite number of times, not media which wears out after 10k ~ 100k writes to the same area (typically 1M or larger block size), even if only a single bit is changed in the entire block. It has to do with the fact that writing to NAND-based flash memory adds charges to MOS transistors, which, by design, have characteristics of capacitors, and unfortunately, what makes them retain data with power off ALSO results in undrainable permanent charge accumulation every time a 0 is written to the bit (the slowly accumulating charge can't even be reversed by writing 1 to the same bit location)

This categorically rules out ext4, xfs, and jfs.

Don't use F2FS or UBIFS. F2FS is meant for raw MTDs with no wear leveling and UBIFS is meant specifically for UBI devices with no wear-leveling.

USB sticks and the various sizes of *SD cards all have wear leveling because they're "consumer" products made for people whose computer expertise starts and ends at using their cell phones and copying pictures from one office computer to another.

BTRFS appears to currently be the least damaging to those NAND-based devices which have their own internal wear-levelling such as USB sticks and *SD cards, by limiting the rate that it writes to the storage device, that being said, BTRFS still isn't entirely stable.

I don't know where "SSD drives" manufactured with SATA interfaces fall in this realm.

Me? I leave only the absolute essentials on my RPi's microSD card, and I get as big a card as I can find (currently 256G size), and I moved the swap and as much of the filesystem as possible onto a couple partitions on a 1 TB 2.5" laptop drive, connected through a powered USB3.0 hub (to keep from burning out the controller chip for my RPi's USB 3.0 port by offloading the current consumption to a separate power supply).

If the RPi4 would let me boot directly off of a USB port (Correction, see below at the end of my response) like RPi3 does, I would have the entire OS installed on the disk like I did with RPI3 (but unfortunately, RPi3 only has USB 2.0, which is a painfully slow way to access mass storage, and RPI3's have only 4GB, even though the CPU cores are aarm64 architecture, eliminating a significant portion of the benefit of 64-bit address space.)

This is not definitive, it's merely the best my research turns up at the present (Feb 2022).

See here for F2FS and what type of solid state memory it's designed for and what hardware it shouldn't be used on: http://www.linux-mtd.infradead.org/faq/general.html

See here for UBIFS and what type of solid state memory it's designed for and what it shouldn't be used on: http://www.linux-mtd.infradead.org/faq/ubifs.html

CORRECTION: rpi-imager now has a method for changing your RPi 4 and RPi 400 to boot from USB (and also boot from network): https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb

1
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Mar 1 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.