0

I have a Rasperry Pi on which I run Ubuntu on an SD card. The setup runs „headless“, meaning I had installed Ubuntu on the SD card on a separate pc and then configured the rest by SSH-ing into the machine.

Last time I connected via SSH, because some apps I ran on the machine were terribly slow, I tried „apt get upgrade“ and got the message back that my system was a „read-only“ system.

Some digging on the internet lets me believe that Ubuntu switches to read-only when there is a problem with the file system.

What I did then is to try rebooting the system, with the effect that it didn‘t boot properly and now I can‘t even ssh into the machine anymore. Since the Ubuntu runs headless, I also can‘t fix it by attaching keyboard+monitor to it.

Hunch is that my only chance is to fix the system by mounting the SD card on another system.

Here‘s my questions:

Any ideas what the best strategy would be to fix my Ubuntu, so that at least I can ssh into it again (preferrably without having to set up Ubuntu from scratch) Any ideas how I can find out why my system went into read only mode, so that I can prevent this in the future? Many thanks for your all‘s help!

When I had this problem already once, I reformatted the SD card and reinstalled everything. But this can‘t possibly be the best solution. I‘d like to understand how to best „cure“ this, and how to prevent it in the first placeP

1
  • TL;DR "headless" doesn't prevent you from attaching a "head". In other words, you currently run it headless because you choose to do so ... there's nothing stopping you from attaching a keyboard and monitor (except maybe a lack of keyboard, monitor, appropriate HDMI cable) to get a better understanding of what the issue is and possibly fixing it Commented Jun 15, 2023 at 1:33

2 Answers 2

1

Some digging on the internet lets me believe that Ubuntu switches to read-only when there is a problem with the file system.

Typical syndrome of damaged storage device.

That's common on RPis. The reason is simple: Consumer SD cards are meant for writing photos and videos to, as fast as a human can take photos and videos, and not for systems writing logs, databases and configuration files.

You'll notice that's the case by looking into dmesg and looking for mmc-related errors.

The controller in the SD card does what it can, but in the end, this usage pattern wears out some SD card's physical storage rapidly, and there's nothing that wear leveling can do if the amount of writing is high enough.

If that's the case, not even mounting the SD card externally will restore its ability to write data.

However, not attempting to write to it any more in your RPi is a very good idea if you want to retain any of the original data.

When I had this problem already once, I reformatted the SD card and reinstalled everything. But this can‘t possibly be the best solution. I‘d like to understand how to best „cure“ this, and how to prevent it in the first placeP

There's no cure; the only option would be to get SD cards that are designed for this usage pattern, and that's hard. Even the legitimate producers package wildly different chips as the same product, and there's a lot of greymarket and counterfeits, even in western retail markets.

The best solution is to make sure you have a quick way to assemble exactly the SD card you need; a common solution for that is indeed just installing a base Ubuntu + SSH server on an SD card, then take an image. This will be your base for future setups.

In order to avoid rapid degradation of the SD cards of your future, I'd recommend the following:

Even a fast SD card is slow in a Raspberry Pi. Even the best SD cards in the newest RPi 4 benchmark at less than 4 MB/s random writes. That's laughably little for modern flash memory! Your RPi's USB2 (in case of the RPi4: USB3) is much faster than that. Selecting to store everything but the /boot partition on external (I think that can nowadays be done during debian installation, not sure! Even if you have to move the / volume to external storage manually, change /etc/fstab and regenerate the initrd, you'd only need succeed with that once: you image that state and use it as base in the future!), USB-connected memory will make your system run faster. Extremely fast USB storage is surprisingly cheap – with desktop SATA SSDs above 100GB running below 10€, and external USB cases for these below 10€ as well, there's no reason to rely on write-averse media like SD cards if you have storage-intense workloads. But: even a medium-grade USB "thumb drive" will be fast – but beware large quality differences, just like with SD cards, and even more bottom-of-the-barrel devices in the least-cost department.

No matter whether you decide to move the most of your system to USB storage or not, you'll also want to make sure that whatever you do to your system can be replicated with a single call, instead of you spending hours on installing the same things again. Ansible is a common tool for that – you write a "playbook" that says which apt packages to install, which files to copy over from your PC, which users to add, which configuration files to modify… and it just goes off and does that to a freshly set up system. All it needs is SSH access to that system.

2
  • Because you suggest USB storage, in my experience some USB thumb drives, although quite fast when writing one single file, do start to behave very strangely when confronted with OS workloads of consecutive reading and writing. They seem to lock up and everything gets super slow, kernel complaining about 120 second timeouts etc. But if you bring up the patience and let apt run for hours, it eventually will finish. Maybe buggy controllers, maybe they do that on purpose to prevent that use case. Never seen that kind of behaviour with SD cards. But yes, they die quickly!
    – xenoson
    Commented Jun 15, 2023 at 13:23
  • @xenoson exactly what I meant with "beware large quality differences". Both the USB mass storage device specification as well as the MMC interface (which is what SD cards use) allow the controllers in the actual storage device to just arbitrarily buffer, and demand breaks (same is true for SATA and nvme, by the way). It's just that on expectation, external USB-to-SATA controllers are somewhat OK speedwise, and the actual buffering / IO happens in the SSD, so that the SSD-via-USB-converter solution often works very well. But as you say, there's also big differences in speed and reliability! Commented Jun 15, 2023 at 13:41
0

First of all, many thanks for taking the time and providing the helpful pointers.

I was able to solve my problem by connecting a monitor to my Raspberry Pi, and then access the console to run a fsk-command. The trick was to connect the monitor before the machine was running (hot-plugging did not work), which left me under the wrongful impression that "headless" systems don't even offer a console (where in reality they do, which is where i could tackle my problem).

What I also learnt here is that keeping the applications on the SD card is a bad idea, and I will look to move parts of my system onto a USB drive and keep the core OS on the SD card in a read-only mode.

Thank you all

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .