1

I'm using a custom buildroot Linux kernel for Raspberry Pi, based on the Pi kernel 5.10.92 (64-bit). I've flashed this image onto micro SD cards from two different manufacturers and installed them in a custom Pi CM4 PCBA.

After several weeks of use, some SD cards have become unbootable. When I insert the SD card into a Linux machine and run dmesg, it displays the following:

sd 2:0:0:1: [sdd] 61069312 512-byte logical blocks: (31.3 GB/29.1 GiB)
sd 2:0:0:1: [sdd] Write Protect is on
sd 2:0:0:1: [sdd] Mode Sense: 23 00 80 00
sdd: sdd1 sdd2 sdd3 sdd4

Some things to note: Due to the use case of the hardware, the OS automatically reboots every 12 hours U-boot is the target of the pi’s bootloader to enable a dual-partition (A/B) failover scheme. It checks it’s environment (CONFIG_ENV_IS_IN_MMC=y), CONFIG_ENV_SIZE=0x4000, CONFIG_ENV_OFFSET=0x1000000) The image layout is as follows: First 4 MB are reserved for u-boot’s environment Boot partition (fat32) Rootfs A (ext4) Rootfs B (ext4) Persistent data partition (ext4)

When I tried mounting each of the partitions individually on another Linux system, the boot partition (/dev/sdd1) mounted as read-only. For the other three ext4 partitions, dmesg displayed:

EXT4-fs (sdd2): INFO: recovery required on readonly filesystem
EXT4-fs (sdd2): write access unavailable, cannot proceed (try mounting with noload)

Even when mounting with the noload option, the device is still reported as write-protected and mounted read-only, like the FAT32 boot partition.

I'm trying to understand what might have caused the entire block device to become read-only. Currently, I haven't found a consistent method to reproduce this issue on more SD cards, so any suggestions for reproducing the problem would be appreciated. We have other SD cards that have been in use for the same duration or longer without any issues.

Improve this question
4
  • It all points to file system corruption. Perhaps use better SD cards?
    – ChanganAuto
    Apr 3 at 23:42
  • Experienced the exact same issue, and in my case it indeed turned out to be flaky SD cards. You don't state which Pi you have, but if it's Pi4 and you're rebooting the system twice a day, I'd suggest booting from USB disk/stick, as those are more robust than an SD cards.
    – Peregrino69
    Apr 4 at 6:42
  • Is there a hardware switch at the SD card which is in the wrong position?
    – Wolfgang Wilke
    Apr 4 at 7:09
  • It's worth noting that not all SD cards are alike in terms of shelf-life. Some products exist that are designed to handle large continual write. Just for example (not promoting it) Sandisk have a "High Endurance" range. While they brand this for dashcams and similar, the same features are also useful for safely running linux.
    – Philip Couling
    Apr 4 at 11:17

1 Answer 1

Reset to default
2

It seems that the memory controller in your SD card has decided it's not going to allow you to write to the medium anymore.

The very most likely reason for that is that it ran out of blocks to wear-level:

Flash memory is essentially a large array of capacitors which you can individually either leave at their uncharged state, or charge. Whether or not a charge is stored in the capacitor signifies a stored bit.

To "uncharge" a capacitor, you need a higher voltage.

SD cards consist of NAND-style flash memory, which only supports a block-erase operation. This leads to a write operation to, say, a single byte, looking like this (I have a way deeper explanation here):

  • Read the full block in which that byte is located into some SD card-internal RAM
  • Erase the whole block of flash
  • Modify the byte within the block in RAM, and
  • Write the result back to the block in flash

Now, the problem is that these erase/write cycles actively degrade the memory cells. You can't do that infinitely often to the same cell! Sadly, file systems tend to do exactly that. There's a definite block in which my file system stores the names, dates and properties of files, and my programs will probably be writing to specific file positions more often than to others. We're stressing the flash memory!

Now, there's an easy trick to be done here: Notice how we have to erase the whole block? At that point there's no actual advantage to re-using the original block. Any other erased block is just as good. So, we just use a block that hasn't been written to as often. Then we mark in a table that this is where the logical block is stored.

That wear leveling drastically extends flash memory device life time. But: it has its limits. If you regularly write large amounts of data, like logs, backups, databases (especially those where you're updating small blocks of storage! No matter how small, that's a complete block update, unless you cache that), at some points doing more writes will lead to the capacitors becoming "leaky" enough (the readout voltages becoming unreliable enough) that you will just lose the data you've stored. Your SD card controller is nice and saves you from that – it's worse to write data and lose that data than to know you can't save data.

You probably want to buy new SD cards. Going forward, a couple of things you can do:

  • Reduce write stress
    • Disable file access time saving in file systems
    • Don't write logs that you don't need to storage
    • random-overwrite / steady-update storage belongs on a SSD attached via USB, not the system boot SD card
      • I'm totally aware of you saying this is CM PCBA, so "just add a USB device" is probably more of an investment than the ca 30 € a 240 GB external SSD costs.
  • Give the memory controller freedom
    • If this was a SSD attached via nvme or SATA, you could, after operations that yield free space on your ext4 file system, use fstrim to tell the flash controller which blocks on the SD card can be considered unused. These go straight to the pile of blocks ready for reuse.
Improve this answer
9
  • @PhilipCouling I assure you they do. If they hadn't, they wouldn't be cheap. The ability to use very imperfect storage is what makes flash so affordable. (the ability to wear level is a very cheap side product of the ability to remap degraded blocks, which is what happens at factory initialization and progressively during usage; if you need that table that maps "logical" blocks to "physical" blocks, adding wear leveling is not a hard thing.)
    – Marcus Müller
    Apr 4 at 10:49
  • 1
    You're sure right, if I was scraping the barrel for margin on SD cards I produce, then I'd underprovision pool space. But: If you want to talk economically, if you take two cards that have the same reliability after wear leveling, then the cheaper one will be the one with more phantom memory! That might feel counter-intutive at first, but it's not that surprising with a bit of math: memory cost scales with 1. amount of memory 2. inverse of defect rate. It scales linearly in 1., typically superlinearly in 2., but for simplicity's sake, let's assume it's proportional to both.1/2
    – Marcus Müller
    Apr 4 at 11:16
  • 1
    Now, If I have some process that achieves a block defect rate (i.e. how likely any block is to be defective at production) p, then the probability that any given block is intact is (1-p). Then, the probabilty that N blocks are intact is (1-p)^N; playing with numbers, e.g. p=10⁻⁶, N=10⁷, you'll see that this probability is very small very quickly. Decreasing p does help, but just putting in 1.5 times as much memory– works just as well, comes with lower risk of having to sort out anything that was actually worse than expectation in practice, and is supported by your controller hardware anyways
    – Marcus Müller
    Apr 4 at 11:21
  • 1
    ... thats why I put my money in cards branded as long life from big brands.
    – Philip Couling
    Apr 4 at 11:25
  • 1
    yep, good bet. There's a lot of scammy behaviour in the SD market – a consumer buying an SD card for their phone to store more photos, music and movies will not be coming back to the same vendor, no matter how great the product was, anyway, any time soon, and that gives way to all the wrong kinds of competition (sadly, that includes middlemen selling counterfeit SD cards, even at "big label stores). I found bunniestudios.com/blog/?page_id=1022 to be an interesting read.
    – Marcus Müller
    Apr 4 at 11:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .